Fraud detection is a crucial data-mining task in the fields of finance and social media. Traditional machine-learning approaches predict risk based solely on the features of individual nodes. Recent advancements in graph-based methods allow for the consideration of features across related nodes, enhancing predictive accuracy. Especially, Graph Neural Networks (GNNs) have shown high performance on graph-based fraud detection tasks. However, it presents significant challenges to performance and efficiency due to the complex and heterogeneous nature of social networks. This paper introduces a novel approach for fraud detection using a Relation-Aware Heterogeneous Graph Neural Network (RHGNN) model, which efficiently handles the intricacies of input data represented as heterogeneous graphs. Our model leverages a computation graph pre-process and hybrid propagation scheme that integrates both features and topology information for GNNs, allowing for precise and scalable fraud detection. Specifically, we first use a relation-aware node map-reduce to preprocess the computational graph. Then we use the hybrid propagation scheme, which optimizes the collection of neighborhood nodes with reduced complexity and remains the fraud pattern on graph data. This is achieved by alternating the focus between the 1-hop neighbor and 2-hop neighbor in the input graph, thereby enhancing the model without the typical computational overhead. We employ Stochastic Projection Reduction to manage feature dimensionality effectively, ensuring that the model remains efficient even with large-scale graph data. Experimental results on various datasets, including Amazon, Yelpchi, and T-Finance, demonstrate that our model outperforms existing methods in terms of fraud detection accuracy.