Category: Fintech News

Following its recent support for MSCI’s Hong Kong Investment Risk Summit as the event’s exclusive partner, Sunline is extending the discussion to the practical capabilities firms need as they prepare for Type 11 and build OTC derivatives businesses in Hong Kong. As the market continues to develop, Type 11 readiness is prompting securities firms to reassess their data foundations, risk management, trade reporting and operating frameworks. For firms planning to launch or expand OTC derivatives businesses, readiness is not simply a licensing or reporting exercise; it requires coordinated development across data, risk, finance, operations and technology.

Drawing on its technology and project experience in securities and capital markets, Sunline has developed five core capabilities spanning data governance, risk capital measurement, Financial Resources Rules (FRR) reporting, Hong Kong Trade Repository (HKTR) reporting, and margin and collateral management.

1. Build a unified data foundation for granular data management

OTC derivatives businesses rely on transaction, position, valuation, counterparty, margin and collateral data, often distributed across multiple business and management systems. Differences in data standards, data quality and management conventions can create inconsistencies across the operating chain.

Firms need stronger data governance at source to improve consistency, completeness, accuracy and traceability, providing a reliable foundation for risk measurement, financial resources management and trade reporting.

Sunline’s data governance and unified data foundation capabilities cover:

• Data standards definition

• Data-quality controls

• Data mapping

• Data lineage

• Historical data reconstruction

By establishing consistent data standards and governance mechanisms, Sunline helps firms reduce front-to-back data discrepancies and create a more consistent foundation for complex risk measurement and reporting workflows.

2. Strengthen risk capital measurement across multiple risk dimensions

OTC derivatives businesses are exposed to market risk, counterparty credit risk, credit valuation adjustment risk, concentration risk and liquidity risk.

Sunline’s risk capital measurement engine supports relevant risk models and calculation approaches, including:

• Standardized Market Risk Approach (SMRA)

Built-in standardised calculation models cover specific and general interest-rate risk, equity risk, gold risk, commodity risk, continuous options, non-standard derivatives and concentration risk, supporting market risk capital charge calculations.

• Standardized OTCD Counterparty Credit Risk Approach (SOCCRA)

Covers counterparty credit risk charges, margin shortfall charges, credit valuation adjustment (CVA) risk charges, counterparty concentration charges and liquidity adjustment charges.

• Internal Models Approach (IMA)

Supports eligible firms in using internal models for risk measurement and incorporates independent price verification (IPV) to strengthen valuation independence and reliability and support fair-value measurement requirements.

Bringing these approaches together enables firms to strengthen risk and capital management across their OTC derivatives businesses.

3. Enhance FRR reporting and financial resources management

FRR returns draw on risk, transaction, financial and capital data, including quick assets, ranking liabilities, paid-up share capital, tangible capital, OTC derivatives exposures, deductions and risk weights.

Sunline’s FRR reporting module can automatically collect and calculate key indicators, map complex fields for OTC derivatives exposures, deductions and risk weights, and generate FRR returns in the prescribed format.

Bringing data collection, calculation, mapping and report generation into a single workflow can shorten reporting cycles, reduce manual errors and improve consistency in financial resources management.

4. Establish lifecycle HKTR reporting capabilities

OTC derivatives reporting extends beyond trade inception. Terminations, corrections, valuation updates and collateral changes also need to be processed and reported throughout the trade lifecycle.

Sunline’s HKTR reporting solution integrates with trading and valuation systems to standardise, cleanse, map and submit transaction data across:

• Trade inception

• Terminations

• Corrections

• Valuations

• Collateral reporting

The solution supports the generation of ISO 20022 XML messages in line with applicable HKTR technical specifications and provides automated resubmission and exception alerts for failed submissions.

Automated data processing and exception management can improve the completeness, accuracy and operational control of trade reporting.

5. Integrate margin and collateral management to improve capital efficiency

Margin and collateral management spans initial margin, variation margin, margin calls, reconciliation and dispute management. It is an important component of OTC derivatives risk management and capital utilisation.

Sunline’s margin and collateral management module incorporates the Standard Initial Margin Model (SIMM) to calculate initial margin across the following risk classes:

• Interest rate risk

• Qualifying credit risk

• Non-qualifying credit risk

• Equity risk

• Commodity risk

• Foreign exchange risk

The solution covers initial and variation margin calculations, margin calls, reconciliation and dispute management, while bringing margin requirements and capital utilisation into a unified risk view.

Integrating margin, collateral and capital data gives firms a clearer basis for balancing business growth and capital efficiency.

From point solutions to long-term business capabilities

Type 11 readiness should not be treated as a standalone licensing project. It requires coordinated capabilities across data governance, risk capital measurement, financial resources management, trade reporting, and margin and collateral operations.

Starting with a unified data foundation and progressively connecting risk capital measurement, FRR reporting, HKTR trade reporting, and margin and collateral management can help firms build a more consistent, traceable and scalable operating framework for OTC derivatives.

Sunline will continue to draw on its technology and implementation experience in securities and capital markets to support financial institutions in strengthening their data, risk, reporting and operational capabilities.

For banks modernizing core platforms, successful data migration is not simply a matter of getting converted code to run. The true operational risk lies in data integrity- guaranteeing that data assets like financial balances, historical transactions, and customer records remain precise, consistent, and fully traceable across environments. 

Traditional data validation frequently relies on surface-level record counts, manual SQL queries, and fragmented sampling. These manual methods, which heavily rely on individual engineering experience, often fail to detect field-level corruption and trace hidden risks, leaving financial institutions vulnerable to silent data discrepancies discovered only after production cutover.

Addressing this structural gap, Sunline has standardized and automated its Data Migration Agent to encompass a closed-loop mechanism that combines multi-dimensional validation, end-to-end audit, and risk visibility that enables data migration results to be controllable, verifiable, and credible throughout the entire process.

A Governed Migration Architecture

Sunline enhanced its Data Migration Agent by replacing manual verification with a structured, four-pillar validation architecture: 

Dual Validation Mechanism

The platform applies row-count validation as an initial filter, followed by MD5 hash-based fingerprint comparison. After concatenating fields to be validated by primary keys, the engine calculates row-level MD5 hashes to pinpoint discrepancies down to specific tables, keys, and fields, generating exportable reconciliation reports without manual validation SQL scripting.

Stage-Gate Governance and Independent Audit Trail

Migration is orchestrated through a strict nine-stage pipeline, and each phase operates under automated state-machine controls. Each stage has an independent gate. If a required check fails, the workflow will not progress to the next stage. Upon completion, an independent audit module reviews the complete evidence chain, including converted SQL, execution logs, validation results, and gate records. Cutover approval is thus based on verifiable evidence rather than subjective judgment, ensuring comprehensive audit trail coverage.

Automated Exception Remediation

The engine parses SQL compilation logs and error outputs to execute rule-based script adjustments and scenario-specific retries, preserving a fully auditable recovery history while reducing manual intervention.

Proactive Risk Classification

Conversion outputs are automatically tagged into four risk tiers (OK/LOW/MEDIUM/HIGH) to help engineers prioritize high-risk items. Through fingerprint comparison, execution failures or reconciliation gaps trigger blocking controls accompanied by specific error codes and remediation guidance, surfacing risks well before production cutover.

From Code Conversion to Migration Assurance

While the SQL Conversion resolves code execution, the closed-loop data validation and audit mechanism ensures data accuracy. Sunline’s Data Migration Agent fundamentally alters the economics and risk profile of core modernization projects. 

Based on controlled implementation benchmarks, this automated migration framework delivers verifiable operational improvements:

• Up to 90% Reduction in Migration Jobs: Automated field mapping, rule conversion, and script generation streamline development workflows.

• Up to 80% Acceleration in Data Reconciliation Efficiency: Replace manual sampling and verification with automated source-to-target cross-table comparisons.

• 100% Audit-Ready Evidence Coverage: Retain stage-gate logs, validation results, and execution metrics automatically to generate a complete, traceable chain of custody for compliance and IT audit teams, requiring no manual intervention.

• End-to-end Risk Mitigation: Stage gates, dual validation, and audit tracing isolate data issues in advance, protecting operational resilience during platform cutover.

Data platform modernization is not merely a technology upgrade – it is a profound exercise in risk management. By ensuring that every deterministic conversion and migration outcome can withstand rigorous row-level reconciliation and business verification, Sunline’s Data Migration Agent empowers banks to execute complex transformations with trusted verification and full-process traceability in today’s highly regulated financial landscape.

As financial institutions advance platform modernization and sovereign cloud strategies, migrating enterprise data warehouses from legacy proprietary engines to open-source and native database architectures has become an operational priority. However, data warehouse migrations frequently stall during the code adaptation phase. High data volumes, intricate business logic, and heterogeneous syntax differences create extended execution timelines, significant manual remediation, and operational risk.

Sunline structures its data warehouse solution as an end-to-end engineering pipeline spanning three core phases: Program Conversion, Data Validation, and Audit Tracking, among which Program Conversion represents the most complex and resource-intensive bottleneck, as it requires adapting and refactoring extensive business logic across heterogeneous database engines

A Three-Pillar Pipeline for Trusted Database Migration

A successful database modernization effort requires a structured, multi-phase methodology to guarantee operational continuity and data integrity:

The Core Challenge: Data Migration Requires More Than Data Transfer

Heterogeneous database engines possess inherent dialect and structural differences. Traditional database migration tools and generic LLMs are not designed to address the complexity of financial database environments, including dynamic SQL, transaction controls, exception handling, UNION type inference, and vendor-proprietary syntax. Without deterministic execution and contextual memory, they often produce unreliable conversions, leaving engineering teams to resolve manual code issues and identify hidden logical errors during testing.

To address these migration bottlenecks, Sunline has introduced its proprietary SQL Conversion Agent – an AI-engineered, rule-driven automation platform designed to streamline complex SQL and stored procedure translation for enterprise financial databases, which compresses project execution timelines and delivers unmatched efficiency without compromising engineering quality.

Technical Capabilities of Sunline SQL Conversion Agent

Compared to the unpredictability of generic large language models and the adaptation of legacy pattern-matching tools, the SQL Conversion Agent delivers higher stability, domain specialization, and operational control:

• Abstract Syntax Tree (AST) Parsing: Bypasses superficial text replacement by constructing a structural AST representation of source scripts. The engine accurately identifies nested functions, cross-database references, and procedural control flows, preventing syntax corruption.

• Modular Skill Packaging: Standardizes data-type mappings, function conversions (e.g., mapping NVL to COALESCE), and syntax transformations (e.g., converting DECODE to CASE WHEN) into auditable rule packages. Shared syntax rules are reused across workloads, while isolated edge cases are addressed through dedicated sub-routines.

• Automated Risk Profiling and Traceability: Every converted script passes through an eight-stage Standard Operating Procedure (SOP) pipeline. The agent generates a comprehensive four-tier risk report (OK, LOW, MEDIUM, HIGH), allowing engineers to focus manual intervention strictly on high-risk items.

Accelerating Modernization Timelines While Eliminating Silent Errors

By replacing manual review cycles with an automated, deterministic translation engine, the SQL Conversion Agent resolves the primary bottleneck in data warehouse modernizations:

• Timeline Compression: Automates up to 99% of syntax adaptation details, reducing overall migration timelines from months to weeks or days.

• Risk Mitigation: Early risk profiling shifts issue detection to the pre-deployment phase, preventing silent logic failures in production data pipelines.

• Resource Efficiency: Minimizes manual code refactoring costs, allowing enterprise data teams to focus on platform architecture, data governance, and analytics delivery.

Establishing a deterministic program conversion mechanism forms the foundation for secure database modernization. By replacing manual trial-and-error with an AI-engineered, rule-guarded pipeline, Sunline transforms high-risk code refactoring into a fast, repeatable, and fully audited enterprise capability.

As financial institutions pursue more granular, data-driven management, democratizing self-service data access across business lines has become a key operational priority. However, enterprise data teams face persistent friction with traditional ad-hoc retrieval models. When banks attempt to deploy conversational AI or generic Text-to-SQL solutions to accelerate data access, they frequently run into significant structural barriers:

• Algorithmic Hallucinations and Query Inaccuracy: Direct, unconstrained translation of natural language into executable SQL yields low accuracy when applied to complex banking schemes.

• Fragmented Metric Definitions: Core key performance indicators (KPIs) and business definitions often reside in implicit logic, preventing AI models from resolving true business context.

• Governance and Security Risks: Black-box SQL generation risks producing field disorders, invalid table joins, or unauthorized data access.

• Data Heterogeneity: Enterprise data remains distributed across diverse, multi-platform database environments, creating navigation barriers for business users.

The Solution Architecture: A Three-Tiered Semantic Foundation

Sunline is addressing challenges by introducing ChatSQL, a natural language querying component built on the DataMind platform. Rather than relying on the direct, black-box natural-language-to-SQL translation, ChatSQL innovatively introduces a semantic layer between natural language processing and SQL generation, allowing field definitions, metric logic, table relationships, and access rules to be explicitly modelled before a query reaches the underlying databases. The platform operates on a three-tiered deduction model:

Natural Language (NL)→Semantic Query→Executable SQL

Under this model, ChatSQL established an end-to-end governed query pipeline: 

Natural Language Input → NL-to-MQL Parsing→ MQL Compilation →SQL Generation → Database Execution → Result Return 

As an intermediate abstraction layer, MQL isolates physical database complexity and disparate table structures, enabling business personnel to query specified metrics without knowledge of underlying data schemas or physical storage locations. It also embeds unified metric definitions, SQL injection protection, and field-level access control within the semantic tier, applying validation at the point a query is initiated and balancing ease of use with the data-security requirements of financial scenarios.

Governance Built into the Query Lifecycle

ChatSQL’s semantic model is maintained as a two-layer knowledge engine:

• Definition Layer: Standardizes metric formulas, target business contexts, and contextual relationships.

• Mapping Layer: Maps business definitions directly to underlying physical tables, database attributes, join pathways, and filter parameters.

As definitions are updated through configuration rather than code, business experts can participate directly in metric governance.

The platform agents invoke these semantic services through the Model Context Protocol (MCP), connecting intent recognition to SQL generation, with precise semantic matching, field-level permission isolation, automatic service fallback, and canary release, ensuring every query is evidenced and fully traceable.

Financial-Grade Controls by Design

To maintain data integrity and regulatory compliance, ChatSQL enforces a five-stage workflow with each stage’s outputs isolated and retained for audit: 

If query ambiguity or conflicting metric pathways are detected, the system prompts the user for clarification rather than allowing unverified assumptions. 

This workflow is reinforced by five financial-grade security safeguards:

1. Semantic-Restricted Generation: All SQL outputs are strictly generated from the semantic layer; direct reverse-parsing of raw data tables is prohibited.

2. Serial Execution Enforcement: The five-stage pipeline operates in a non-bypassable serial sequence.

3. Schema Boundary Interception: Eight standard query validation rules intercept fields outside the established semantic model.

4. Pre-Execution Verification: Enforces read-only database locks, query statement limits, credential isolation, and database compatibility checks prior to execution.

5. Post-Query Compliance Audit: Conducts a six-dimension audit on query outputs before rendering visual charts to end users.

Turning Self-Service Analytics into an Operating Model

In live production deployments, ChatSQL delivers validated operational performance enhancement with 95.2% query accuracy across core natural language analytics scenarios in banking operations, alongside an 80% improvement in self-service data retrieval efficiency. 

The significance extends beyond query speed and accuracy. By enabling business teams to retrieve standardized metrics independently, ChatSQL reduces routine data requests. Data engineers can consequently move from repetitive ad-hoc extraction toward data governance, data modelling, and deeper analysis.

The broader direction is a shift from simply making data available to establishing a governed framework through which more personnel can use it safely and consistently. For banks building AI-ready data foundations, the combination of semantic modelling, controlled query execution, and self-service analytics provides a practical layer between enterprise data infrastructure and business users.

ChatSQL represents an application of Sunline’s broader DataMind architecture: using AI not simply to generate outputs, but to connect business context with governed data execution.

Testing has always been an important step to ensure the quality of the software R&D process, but with traditional R&D model, there has been a lack of linkage between testing and the software life cycle. With the promotion and implementation of the DevOps model in the software industry, more frequent delivery further aggravates the industry’s concerns about testing. Inefficient testing is often the primary reason for delays in delivery, and the testing process has become the biggest bottleneck for enterprises to transform into DevOps.

In order to solve such challenges, the concept of “continuous testing” was proposed and gradually became the inevitable pursuit of the industry. In DevOps, continuous testing provides a continuous feedback mechanism that acts as a catalyst throughout the product delivery pipeline, with automatic feedback at each stage ensuring that defects are addressed early in the development process.

This article will focus on sharing how Sunline’s R&D collaborative management platform helps companies get feedback on software release business risks as soon as possible in the “continuous testing” stage of DevOps.

Building an integrated continuous testing platform with technology accumulation

As a carrier of DevOps practices in the testing field, continuous testing is a basic and continuous activity that should run through the entire software delivery cycle. Compared to the traditional testing mode that is prone to many issues in the DevOps era, the first thing that continuous testing needs to change is the “post-testing” situation. Emphasizing on pre-testing, continuous testing define tests as early as possible with parallel testing and development, maintaining close collaboration in the process to achieve quick feedback on business risk.

Focusing on the ultimate goal of “improving efficiency through continuous testing”, MOne Test, an integrated continuous testing platform created by Sunline’s MOne product team, helps the R&D team take testing as a basic activity throughout the entire software delivery process with powerful test management functions. In the process, the software delivery cycle is greatly shortened, testing and R&D are synchronized and iterated, continuous testing is realized, and high-quality delivery is achieved through:

·       Full life cycle covering different stages from test planning, test execution to test report analysis.

·      Rich team collaboration supporting testing teams of different sizes, and provides all the functions to meet the needs of team collaboration and process automation.

·       Continuous testing with the ability to integrate the testing process into continuous delivery and DevOps system, seamlessly connecting to defect management tools (JIRA, MOne Project) and continuous integration tools (Jenkins, MOne Pipeline), etc.

·      Integration covering test case management\review\execution, interface\WEB automation, fault drill, performance test (compatible with JMeter native jmx script) and other functions.

The practice of continuous testing within iterations in MOne Test

The testing process generally includes use case design, use case review, test execution, and test reporting. MOne Test integrates continuous testing platform to increase the collaboration and feedback between testing and other roles in the process, and opens up data interaction with the project management platform. The purpose is to help the team solidify practical experience through product capabilities, so as to achieve high efficiency within iterations test with:

Efficient organization and management of test cases to easily organize use cases in the tree-structured use case library, testers can flexibly reuse test cases to form use case reviews and test plans, greatly improving work efficiency.

Docking with mainstream project management platforms such as JIRA, MOne Project, etc., test cases can quickly associate requirements and defects, and in the process of test case execution, defects can be quickly created and synchronized to a third-party defect platform.

Rich online collaboration with multiple roles in the team allow multiple participations in the use case review and execution at the same time, the work status is synchronized real-time, cross-test redundancy is avoided and information is synchronized efficiently.

Efficient testing requires automation as the cornerstone

In the high-frequency delivery scenario of DevOps, how can the team improve the efficiency of test execution? The first thing that comes to our mind is automated testing as it is the basis of continuous testing. Only with a high degree of automation can it meet the high-frequency release requirements of continuous delivery.

Currently, most testing tools use open-source interface automation frameworks or open-source interface testing tools to automate interfaces, such as Postman and JMeter. However, this method has high maintenance costs and lacks the function of batch execution, which cannot meet the needs of rapid verification.

MOne Test is an integrated continuous testing platform, which fully refers to the page layout of open-source tools in design. It is easy to use, and supports batch execution, which can realize “zero code” interface automation.

Rich parameter transfers to meet the needs of data construction with the main types of parameters are as follows:

·       Parameter constructor to quickly generate data for testing, such as random numbers, timestamps, ID numbers, etc.

·       Public parameters put public data in the script into the public parameters for unified management, which is easy to maintain.

·      Step parameters allow quick reference of data returned by the previous step, and support referring to the data returned by different types of steps.

·      Transaction data show data reference across scripts to meet complex banking scenarios.

·      Project environment variables allow quick change to the script URL and public data by switching the project environment (test, SIT, UAT) when the script is executed so that the same script can cover different running environments.

Flexible scenario arrangement supports the combination of steps such as API, API template, database query (MySQL, Oracle, PostgreSQL), JavaScript script, etc., to jointly complete the interface test of complex business scenarios.

Support batch execution visualizing the entire execution process of each API request i.e., request header, request body, response header, response body, assertion in the script execution step will be recorded. If the execution fails, the failure step will be prominently marked and the cause of the failure will be displayed.

Integration of automated testing capabilities into the DevOps system provide Jenkins plug-ins, which can directly schedule batch tasks on the test platform after API key authentication. Self-developed MOne Pipeline provides tool-specific operation steps to easily realize one-click automatic deployment and testing.

MOne Test helps the interface automation construction of a banking system

·       The automated scripts cover a total of 2,000 key transactions and develop nearly 3,400 automated test cases.

·     Batch tasks can be fully returned to the system every week and each time before going online, and the system can be fully expanded and executed concurrently in multiple environments.

·       Generate a fixed system start date through transactions, and restore data through scripts after each round of testing performed.

Through the construction of interface automation, the efficiency of the bank’s testing can be greatly improved. Multiple scripts can be executed concurrently during task execution, thereby greatly shortening the regression cycle and reducing the risk of digitalization.

In the process of practicing continuous testing, enterprises not only need technical support, such as the basic capabilities of continuous development, continuous integration, and continuous deployment but they also need to pay attention to the precipitation of data, and continuously optimize behaviors based on data indicators, so as to realize the benefits of DevOps.

In the next article, we will share more on the practices in the “continuous deployment” phase, so stay tuned!

In the previous article, we mentioned that DevOps is a culture that values communication and collaboration between “software developers (Dev)” and “operations (Ops)”. The methodology used is different from the traditional waterfall model, spiral model and other concepts, its core is “agile” which results in automation.

More enterprises expect to achieve enhanced delivery efficiency through the introduction of DevOps model, in order to improve customer satisfaction and create more business value. But at the implementation level, how to successfully practice DevOps is still a difficult problem. The research and development collaborative management platform (MOne) independently developed by Sunline is a one-stop solution covering the whole process from requirements, design, development, construction, testing, release to deployment. Through the platform’s method of standard processes precipitation and agile practices, it forms effective empowerment at all stages of the development process via DevOps.

This article will focus on sharing how the Sunline R&D collaborative management platform helps the R&D team achieve a whole new level of digital collaboration in the “agile iteration” stage of DevOps.

The whole process of agile iteration

The implementation process of agile iteration includes: product team planning the goal → development needs dismantling, scheduling development → testing, testers perform iteration scheduling, development, testing → agile coaches track and feedback the iteration, in this process, the scientific division of resources and excellent collaboration between different teams in China are the keys to truly realizing “agile”.

·       Product team

As a bridge between the analysis of user needs and introduction of research and development needs, it is necessary to define the key features of the product according to the analysis of user scenarios, so that the product made is more relevant to the user.

·       Development team

As for product development, it is necessary to analyze and disassemble the research and development needs of the development team according to product characteristics to focus on the needs of solving pain points faced by users.

·       Testing team

The testing team serve as a quality component in the product development process, from use case management, automated testing and other means to ensure the products delivered to users can withstand different scenarios or tests.

·       Agile Coach

Agile coaches monitor the progress, identify possible risks and issues face during the process, and help the R&D team to continuously improve the efficiency during the agile SCRUM framework.

Agile iterative practice in different scenarios

Scenario 1: Identify user needs based on user scenarios

User scenario analysis is mainly provided to the product team for the deduction and analysis process. In the analysis process, there are two key pieces of information.

1.     User activities: According to the user’s operation and process, the user’s operation process is deduced step by step. We call this process the analysis of user activities. This process is mainly to identify the key scenario behaviors on the user side.

2.     Product capabilities: User activities need the characteristics of products (components) to support them. For example, the “click pipeline release” activity in user activities requires two products such as “dynamic orchestration pipeline” and “pipeline scheduling execution” of the MOne Pipeline component to support.

The user scenario analysis designer serves to carry the user’s scenario analysis process, and finally identify what capabilities of the products (components) are required.

Scenario 2: Focus on the key features of a product

The “capability map” is a form of expression of the product (component) key features from the perspective of the user. The input source of the product’s capability composed of the following two parts:

1.     The ability to identify product components after deducing user activities through “user scenarios”.

2.     Plan which product components the system has in advance, and the key capabilities of the corresponding components.

When all product capabilities are aggregated, a “capability map” of the product can be formed.

Through the “capability map”, we can better manage and track product capabilities.

Management of product capabilities based on R&D requirements: Product capabilities represent the output of product characteristics, and capabilities can only be released after the R&D tasks are decomposed and completed. Through the decomposition ability, we form a “to-do list” of R&D concern downwards, so as to realize the connection between product capabilities, and R&D needs. The research and development to-do items can be integrated into rounds of sprints through agile iterations. During the sprint process, the quarterly R&D requirements will be automatically fed back to the progress of product capabilities to achieve synchronous update of capability progress. Finally, when all R&D requirements under the capabilities are closed, it means the upper-level capability development is completed.

Tracking of product capability progress: You can understand the progress of each product component capability through the product progress tracking trend chart. The progress report of product capabilities will be summarized on a weekly basis to facilitate management to understand the development of the product.

Scenario 3: Collaborative mode for multiplayer development

The “user scenario” of scenario 1 mainly identifies product capabilities through user activities, and the “product capability map” of scenario 2 is to better manage and track product capabilities. The above two scenarios are mainly for product team. When broken down into R&D tasks through capabilities, a “to-do list” of R&D concerns is formed.

To-do item pool: By planning an iteration every 2 to 3 weeks, the to-do items are incorporated into the iteration for development.

Visualized iterative Kanban board: Through the visual Kanban board, it is more convenient for the team to align the progress of iterative tasks and predict risks.

Iterative Kanban board can be divided into different columns according to the process. When developers process tasks, they will move the task cards on the Kanban board to realize the state flow process. When all the cards are in one column, it means that the iteration tasks of this round are completed and the iteration can be closed.

In addition, on the Kanban cards, important and critical tasks can be marked and identified through the “label method”, and “deferred signs” are added to remind members to deal with them on time to avoid delays.

Scenario 4: Driven Development Mode of Feature Branch

The feature branch-driven development mode is mainly to speed up the process of software code development to testing deliverables. We perform offline code coding by checking out the code branch corresponding to the R&D requirements. After the code development is completed, if you want to quickly perform functional verification, you can initiate a merge application process, which will automatically trigger the branch release process in:

1.     CI construction pipeline, which performs source code construction, packaging, quality scanning and other processes, and uploads products to the corresponding product library.

2.     CD release pipeline which automatically trigger the product package to be pulled and deployed from the product library to realize deployment in different environments.

When deployed in different environments, it can be reviewed through development self-test, tester testing, and then to the pre-release environment of the product. After the development of the entire environment is completed, the branch code is automatically merged into the base (trunk) code.

The mode of feature branch development connects multiple roles such as development, testing, and product, which can more quickly break through the barriers of departmental collaboration and speed up the rapid software delivery process.

Through the above four scenarios, we describe the practice of Sunline’s concepts and methods in the “agile iteration” stage of DevOps. In the next article, we will focus on the practice of the “continuous testing” phase. Stay tuned!

In the wave of the digital age, the innovation and development of enterprises depend on whether they can quickly and frequently respond to the real-time user experiences and feedbacks. Those who stand out often have the ability to quickly and iteratively deliver products and respond flexibly to changes. As an extension to the agile concept to the field of operation and maintenance, DevOps provides solid support for enterprises to pursue innovate market responses in the process of digitalization, and is also the driving force for enterprise transformation.

DevOps reshapes R&D operation and maintenance system

The term DevOps comes from the combination of Development (development) and Operations (operations and maintenance), that is, the “integration of development and operations”. This concept emerged in Europe in 2009, due to the pain in operation and maintenance with the traditional model.

For a long time in the development of computer technology and software development industry, R&D, operations and maintenance have been in a clear relationship. From the perspective of R&D, delivering quality products with higher efficiency has always been the demand of R&D engineers while from the perspective of operations and maintenance, computer operations and maintenance technologies worldwide has been replacing manual operation and maintenance with machines and scripts.

The popularity of DevOps grew around 2015 when the IT market started to realize that it can reduce workload and improve work quality by connecting and integrating R&D with operations and maintenance.

Referring to the definitions of top global IT companies, DevOps is not a single technology or tool, or even just a process, it can be understood as a series of tool chains that can develop software at high speed and quality. This model not only improves the efficiency of software development but the performance of the final product has also become the embodiment and application of modern IT enterprises.

Today, with the advent of the digital and intelligent era, software is profoundly changing our way of life. How to quickly and continuously deliver high-quality software to meet the diverse needs of users, so as to reshape the core competitiveness, has become a practical problem that many enterprises must solve. Facing the competitive pressure brought by development and management, more enterprises are introducing DevOps to deal with more complex software development needs and environments and DevOps has been regarded as the key to the future development of enterprises.

Sunline’s exploration in DevOps culture

The essence of DevOps is a combination of cultural philosophies, practices, and tools. Essentially, suitable tools can really improve the collaboration between development, IT operations teams and business teams, and a tool chain with higher integration and more complete ecosystem has become a major trend in this industry.

As a leading software service provider that has been deeply involved in financial technologies for many years, Sunline started the exploration of DevOps culture as early as 2015, and is committed to using self-developed products and various open-source tools to build its own DevOps tool chain and realize an integrated R&D standard system and platform construction. After long-term accumulation of experience in the field of DevOps, Sunline combines agile development methods to create a professional and standardized R&D tool “MOne Platform”, to meet the everchanging industry requirements.

The full name of “MOne Platform” is “R&D Collaborative Management Platform”, integrating the practical experience of R&D processes, implementation specifications, collaborative management and other aspects. MOne builds R&D practice into the platform to guide enterprises in the practice of DevOps. Through the process specification + tool mechanism, MOne supports the entire process from requirements, design, development, construction, testing, release to deployment, forming a closed loop R&D quality monitoring.

MOne has been entrusted with the mission to help teams develop, test and deploy code quickly and independently; deliver value to customers quickly, safely and reliably; as well as effectively improve the productivity of developers. MOne series of products have experienced many business scenarios enhancement, and based on a complete tool chain, it can provide a set of mature R&D efficiency solutions for customers in all walks of life, helping enterprises reduce the construction cost of R&D tools, improve product delivery efficiency, and achieve R&D upgrade efficiency.

Interested to know how MOne series of products achieve integrated R&D, as well as collaborative management in operations and maintenance? Stay tuned to the next few series of DevOps practices articles, and we will unveil them to you one by one.

Since the birth of Sunline API Gateway, it has been applied in dozens of financial customer projects such as Bank of Communications, Bohai Bank, Minsheng Bank’s Credit Card Business, etc. with numerous production tests, displaying high performance and high reliability.

With the development of microservices architecture, the legacy coarse-grained application is divided into many fine-grained microservices. With each service having its own API services and the demands between services have become intricate and complicated, issues such as unified API management, API security, traffic forwarding, and traffic management have become particularly prominent.

Breaking the game—introducing the API gateway

How to solve the problems associated with the development of microservices architecture, an extra layer needs to be introduced between the client and the services as a reverse proxy that initiates request routing from the client to the services which is similar to the appearance pattern in object-oriented design, providing a single entry-point for the API that encapsulates the underlying system architecture, the API gateway.

In short, the API gateway uses a single and unified API entry point to combine one or more internal APIs. The API gateway takes over all ingress traffic and forwards all user requests to the back-end server, uniformly managing the entire life cycle of the API.

The function of the gateway is not as simple as that. The API will also perform traffic governance such as authentication, current limiting, permissions, fuse, protocol conversion, error code uniformity, caching as well as traffic monitoring such as logs, monitoring, alarms, and security precautions such as protocol security, Access security, message security, etc. With the gateway unifying the services, the business side can focus more on the business logic and improve iteration efficiency. Hence, the importance of API gateway is evident.

The API gateway brings multiple values to the microservices architecture system:

• Isolates the external and internal to ensure the security of back-end services.

• Transforms external access control from the network level to the operation and maintenance level, reducing the change process and error costs.

• Reduces the coupling between client and services, enabling independent development of services through the mapping of gateway layers.

• Reduces the frequency of external access and improve access efficiency through gateway layer aggregation.

• Saves cost of back-end service development and reduces risk of going online.

• Provides simple solutions for service fusing, grayscale release and online testing.

• Facilitates application-level expansion.

As the entrance of traffic, the non-functional characteristics of the gateway such as high performance, high availability and scalability are crucial.

Sunline API Gateway relies on its good scalability to continuously improving and enriching its functions to dock with many internal and external systems, expand multiple access and access protocols. The gateway also supports multiple traffic management strategies and provide more comprehensive security defense control.

The following is a detailed analysis of the design practice of Sunline’s API gateway from three aspects: the overall design of API gateway, API governance design, and API security.

The overall design of Sunline API Gateway:

1. Technical architecture design with maximum functional decoupling

At present, the maximum functional decoupling is not achieved. On this basis, Sunline has innovated comprehensively. The gateway service is divided into two parts: the control end and the operation end which are operated separately, so that the gateway operation service is separated. External dependence truly decouples the gateway to the maximum extent.

The API gateway adopts a front-end and back-end separation architecture model, is developed in Java language and uses the current mainstream technology stack Spring Boot and Spring Cloud systems.

• The main function of the control terminal is to manage gateway configuration, UI interaction, push data to the gateway running terminal, etc. The control terminal and the server side have a clear division of labor so that the gateway running terminal that is really responsible for processing the request will strive to maximize resources.

• The running end of the gateway is the gateway service. The core mechanism is the filter chain mechanism, the access and output mechanism. The gateway running end is docked with a variety of basic components including the monitoring center, Registration center, link center, log center, configuration center, etc. In order to ensure that the parameters configured by the user are lost in push, the gateway operation service will also regularly pull data from the gateway management and control service to achieve the effect of two-way data synchronization.

2. Highly expandable design provides more comprehensive expansion

The gateway running side adopts SPI mechanism which greatly increases the scalability of the gateway. In addition to the filter extension and management function extension that are supported on the market, Sunline can also provide access protocol extension, encryption and decryption. The expansion of multiple locations such as mode and message greatly increase the scalability points of the gateway.

• The filter extension function of the gateway belongs to a filter chain in the entire project. Filter of the gateway can be chosen through dynamic configuration of the page, for example the addition of a certain authentication mechanism to expand the filter chain.

• Access extension on the basis of the existing gateway multi-protocol to extend an access protocol, such as Dubbo, TSF, etc.

• Encryption and decryption extension extend new algorithms and rules such as encrypt and decrypt request, response messages and add signature verification.

• Gateway request response secondary extension supports the modification of the access request and the received response at the gateway level.

• Extended gateway response code and response information extension of the gateway response to the code and information is used to adapt to various response code response format requirements.

Except for the filter extension, the other extensions are for inbound and outbound.

3. High-availability design-logical cluster division with easy management and maintenance

The gateway is divided into two services, the control end (data control) and the running end (API call), and they run separately. The running end uses local cache to store information without any read library operations. When the control end is Down, it can still continue. API call.

The gateway server adopts a stateless cluster architecture that can contain multiple gateway instances. The cluster can be classified as a logical instance with each instance corresponds to only one gateway control terminal to prevent data confusion. Compared to the physical cluster division that is commonly used on the market, this logical cluster division is better to manage and maintain.

The gateway control end will actively send heartbeat detection to the running end and the gateway running end will periodically synchronize data to the control end to prevent inconsistencies caused by abnormal data synchronization at the control end.

The client accesses the gateway instance through the load balancer. The load balancer can adopt soft load or hard load mode. The load balancer can use MS architecture to avoid single points of failure.

API governance design:

1. API current limit

Limits the number of times API is accessed, ensuring the service to run normally under pressure and prevent the service from crashing due to excessive traffic. Distributed current limiting is implemented by distributed cache Redis.

When a request enters the RateLimiter Filter, a set of keys will be constructed according to the current request, then judged whether Redis is available. If it is available, Redis will be used for cluster current.          

2. Fuse downgrade

When the service fails, in order to prevent the failure of the entire system, a fuse downgrade strategy is adopted for the system. Fuse downgrade processing can be performed according to dimensions such as the average response time, the proportion of abnormalities in seconds and the number of abnormalities in minutes.  

3. API routing

API routing refers to invoking routing to different back-end services. Gateways support routing based on client IP, ratio, caller and custom methods according to the invocation as well as also support priority configuration.

Four modes of API routing include IP mode, keyword mode, tenant mode and proportion mode.

API security:

1. Protocol security

In order to ensure the security issues in the process of accessing the API, the gateway has added support for https in the design and the access method of https can be used directly to access the API in the gateway.

2. Access security

In many cases, the API is directly exposed to the public network so it is likely to be accessed maliciously. What the gateway has to do is to prevent such malicious access from occurring. Through JTW authentication access, permission control, signature authentication, black and white lists as well as other means to reduce the risk of API will be maliciously accessed.

Compared to a single access security method, Sunline’s access security is more comprehensive in addition to the existing access security as it can continue to expand other security methods:

• Through authority control interface authorized by the administrator, the client has authority to access. If it is not authorized, it will be intercepted at the gateway and the response will be given to the client without access rights.

• Through signature authentication, the request parameters are generated by the SHA256 algorithm | RSA | national secret and other operations to generate a signature value according to the rules. The gateway verifies the customer’s signature and continues after the verification is successful, otherwise it will be directly intercepted.

• Black and white list verify configuration.

• Through JWT authentication, a token from the gateway is applied before accessing the API. Each time the API accesses the Token, the gateway will analyze the Token including Token validity period verification, access authority verification and visitor identity verification, proceeding and intercepting as necessary.

3. Message security

When the client calls the API, the security of the incoming message is very important. The gateway ensures the security of the message by encrypting/signing the message.

• Encryption of the message ensures the security of the message during the access process. In addition to the currently supported AES, DES, RSA and national encryption methods, other encryption and decryption methods can be extended through the SPI mechanism.

• Signing the message to ensure the integrity of the message during the access process in addition to the currently supported RSA, SHA256 and national secret methods. Other signing methods can also be extended through the SPI mechanism.

4. Traffic safety

As an entrance of all applications, the gateway carries the access of massive traffic and the pressure of malicious traffic attacks that may erupt at any time. Therefore, flow control is a necessary part of gateway security to ensure the normal operation of the service and prevent the service from crashing due to excessive traffic.

Summary

In the rich practice of financial customers, Sunline believes that as a portal for enterprise capabilities, API gateways not only have basic request forwarding, protocol conversion, routing, security control and other functions but also high performance and high stability. It needs to have good scalability to facilitate the continuous enhancement of gateway capabilities. During the implementation of the gateway, it is necessary to plan the interaction between the gateway layer and the service layer. The decoupling of the gateway layer and the service layer to facilitate the independence of the work of each team and at the same time, in the management of the API, it is necessary to provide the full life cycle of the API to support management functions such as publishing, configuration, authentication, flow control and monitoring.

Whether it is microservices, distributed architecture or grid service architecture, API gateway is an indispensable part. As the traffic changes between services show explosive growth, API gateway serves as the system entry, playing an increasingly important role in the performance and reliability improvement of the system.