Reconciliation is a supporting system required by any billing system. Regardless of whether reconciliation serves as the primary or bypass payment system, it should be performed during or after payment to ensure billing accuracy. With TDMQ for Apache Pulsar, the reconciliation timeliness is guaranteed, without affecting the critical transaction path.
Encountered Issues
1. System decoupling
Transactions involve numerous systems, which need to be decoupled to prevent mutual impact.
2. Data arrival time gap
There is a time gap for data arrival between systems. Data arriving at different times should be aggregated for calculation.
3. Data consistency
Ensure that data is not lost, preventing abnormal reconciliation results due to data loss.
4. Cross-region data transmission
Systems are deployed across different regions, requiring cross-region data transmission.
Deployment Architecture Diagram
Problem Resolution
We will employ the TDMQ for Apache Pulsar solutions to resolve the issues mentioned above.
1. System Decoupling
To achieve reconciliation between various systems, we could directly report messages to the reconciliation system, which receives messages and performs reconciliation. However, this method faces challenges because the reconciliation system needs to interwork with numerous systems, which increase continuously, resulting in a significant time for cross-system integration and introducing high intrusiveness into system processes on the live network. Clearly, such a design is highly impractical. By introducing TDMQ for Apache Pulsar, systems only need to interwork with the message queue (MQ). In this case, issues in a single system do not affect other services.
2. Data Arrival Time Gaps
Reconciliation requires data aggregation across various systems. To achieve real-time data aggregation, data arrival times should normally be close. However, processing sequences between systems may vary. When data from one system is delayed, we need to control the data reading rate to prevent a large amount of data from entering the reconciliation system and waiting. With temporary message storage of TDMQ for Apache Pulsar, data generated at the same time arrives approximately at the same time.
3. Data Consistency
TDMQ for Apache Pulsar provides highly consistent and reliable data storage to ensure that data is not lost. In addition, it provides highly available services to ensure rapid automatic recovery under exceptions.
4. Cross-Region Data Transmission
TDMQ for Apache Pulsar provides two solutions to implement data replication between multiple cities, providing real-time data replication channels for the business layer.
For data of utmost importance that requires cross-city disaster recovery, TDMQ for Apache Pulsar supports strong consistency data synchronization between multiple regions, ensuring that data is stored in different regions.
For scenarios where strong data consistency is not strictly required, TDMQ for Apache Pulsar provides an asynchronous replication solution between multiple cities to achieve eventual data consistency across regions.
The following table compares the two cross-city synchronization solutions.
Cross-City Solution
Production Time
Disaster Recovery
Storage Cost
Multi-city strong consistency
High
High
Low
Multi-city eventual consistency
Low
Low
High
By introducing TDMQ for Apache Pulsar and real-time computing capabilities, we have transformed transaction reconciliation from a daily mode to a real-time mode, enabling more rapid detection of transaction accuracy.
