Jury v2 Architectural Roadmap

Executive Summary 

The Jury v2 software architectural roadmap focuses on modernizing the technology stack, enhancing performance and scalability, and improving security. The current architecture utilizes Angular v16, .NET 6, and various other technologies, deployed in a containerized environment on AWS. Key initiatives include upgrading to Angular v17, implementing permissions, building out a domain layer, and securing sensitive information with AWS Secrets Manager. The roadmap aims to address technical debt, improve system reliability, and ensure compliance with security standards. 

Current Architecture 

The information below details the technologies used in Jury v2 and describes the environment in which Jury v2 is deployed. 

Technology Stack 

• Front-end:  

  • Angular v16: UI framework developed by Google  

  • Angular Material v16: Prebuilt UI Components developed for use in Angular 

  • Tailwind CSS v3.4.1: Library used to style HTML elements 

  • Authentication:  

    • OpenID Connect (OIDC): industry standard authentication protocol 

    • JSON Web Tokens (JWT): industry standard security token format 

    • Back-end For Front-end (BFF) Encryption Layer: secure client token management library developed by Duende Software  

    • Microsoft ASP.NET Core authentication libraries: latest encryption/decryption/verification libraries provided by Microsoft 

• Back-end:  

  • Microsoft .NET v6 with C# v10 

  • Command Query Responsibility Segregation (CQRS): Separates the concerns of the Command (create, update, and delete) operations from the Query(read) operations 

  • Dapper: An object relational mapping (ORM) library which simplifies the create, read, update, and delete (CRUD) operations on the database by automatically mapping database parameters and data results to C# objects 

  • Fluent Validation:  A library which allows developers to easily implement complex validation rules on all data sent to the API 

  • MediatR: an in-memory bus which implements a publisher/subscriber (Pub/Sub) used to loosely couple the API layer to the Application Layer 

  • NodaTime: A date time management library used to extend the limited date time functionality provided by the .NET framework. Usage of frameworks such as this are key when implementing multi-tenant solutions as tenants and users are dispersed throughout the world and rarely in the same time zone as the servers. 

  • QuestPDF: This is a newly developed open-source library which is used to generate PDF documents using C#. A few reports were built using this technology, but we may phase them out and implement more current versions of the reporting tools (Aspose and ActiveReports) used in Jury v1. 

  • Aspose: Reporting / Document generation library used to build PDF documents from templates which can be defined by users 

  • ActiveReports: A reporting solution used to generate listings or statistical reports 

  • Twilio: This library enables the application to send SMS Messages to Jurors  

  • Authentication:  

    • OpenID Connect (OIDC) - industry standard authentication protocol 

    • JSON Web Tokens (JWT) - industry standard security token format 

    • Microsoft ASP.NET Core authentication libraries – latest encryption/decryption/verification libraries provided by Microsoft  

• Database: Microsoft SQL Server 

• Document Storage: Amazon S3 

Deployment Architecture  

When deployed, the application has the following AWS Footprint:  

• 3 Kubernetes Containers 

1. Front-end: 

   1. OS: Debian Linux v11 

   2. Server: Microsoft Kestrel Web Server 

   3. Frameworks: ASP.NET Core 6.0 

   4. Utilization: Always On 

2. Back-end:  

   1. OS: Debian Linux v11 

   2. Server: Microsoft Kestrel Web Server 

   3. Frameworks: ASP.NET Core 6.0 

   4. Utilization: Always On 

3. Database Updater: 

   1. OS: Debian Linux v11 

   2. Application: .NET Core 7.0 Console Application 

   3. Frameworks: .NET SDK 7.0 

   4. Utilization: Deployment Only 

• 1 AWS RDS Server running SQL Server Engine 

  • 1 Master Database 

  • 1 Database per Tenant  

• 1 S3 Bucket per Tenant for Document Storage 

Assessment of system scalability, performance, and reliability 

• Scalability: Although the application is deployed in a containerized manner, preliminary work has not yet been done to ensure the application can run in a scaled environment (multiple containers). Further research is needed to determine if we may need to implement a framework such as Redis or Amazon ElastiCache to share information between servers. 

• Database Performance: The database structure from Jury v1 has remained largely unchanged through Jury v2. As such, all performance improvements in the v1 database are contained in the v2 database. In addition, the database auditing mechanism utilized in Jury v1 was replaced in Jury v2 which removed over 300 complex triggers. 

• Application Performance: Jury v1 and Jury v2 are both built upon the .NET framework and all algorithms present in the Jury v1 application were ported to the Jury v2 application 

• Database Reliability: Failover Groups will have to be implemented on the database implementation to provide for additional database servers to spool up should the servers hosting the databases fail. 

• Application Reliability: As mentioned above in the scalability section, research is still needed into replicating containerized instances of the application. This work will yield information on what enhancements are required to seamlessly float containers across AWS Availability Zones in case of outages. 

Identification of architectural bottlenecks and pain points 

• While Jury v2 does follow the CQRS pattern, there is no Domain Layer. Business Logic which should be contained within Domain Objects are spread throughout various Command Handlers. This causes duplication of code and is not considered clean architecture. 

• A framework to implement permissions exists in the backend however no code utilizes this framework. This means all API endpoints and all UI screens and features can be accessed by all users regardless of their configured permissions. 

Evaluation of Security Posture

• User authentication in Jury v2 was originally handled by a custom ASP.NET Identity solution implemented within the front-end and back-ends of the application. This custom solution was removed from Jury v2 and replaced with a much more secure integration to Duende Identity Server. This change allowed us to rip out thousands of lines of complex authorization/authentication code from both the front-end and the back-end. 

• Front-End security: Each request for a Web Page must contain an encrypted User Access Token which is decrypted and validated by the BFF layer before being allowed to pass to the Web Server.  

• Back-End security: Each request to an API Endpoint must contain an encrypted User Access Token which is decrypted and validated by the BFF layer before being allowed to pass to the API Server. 

• Jury v2 is regularly monitored by SonarQube and any code security issues are addressed as they arise. 

• Jury v2 will soon also be monitored by Qualys and any security vulnerabilities will be addressed as they arise 

• Database security: Currently the Jury v2 application uses a connection string with a user account containing a username and password to access the database. This should be updated to remove the user account from the connection string. Instead, the API application should be running under an AWS service account and this service account should be given access to the database. This would prevent someone from using the username and password found in the connection string to access the database. 

Review of development and deployment processes 

• Development Process:  

  • Each development task is performed on a fork of the master branch 

  • A Pull Request (PR) is created by the developer for review by a senior team member 

  • The PR is merged into the master branch 

• Development Process Issues: 

  • Due to a lack of deployment environments, work cannot be verified by a QA or UX resource without first checking the code into the master branch. This results in potentially broken functionality being placed into the master branch.  

  • Due to a shortage of resources assigned to the project, Pull Requests require only 1 reviewer instead of the standard 2 reviewers.  

• Deployment Process:

  • Deployment is limited as it currently only deploys to a single environment named Dev.  

• Deployment Process Issues:  

  • Multiple deployment pipelines should be created so different features of the application can be deployed to different environments such as Test1, Test2, and Test3, etc. In addition, Release Candidate versions of the application should be deployed to Preview or Staging environments and analyzed before being pushed to Production. While there is a QA environment that exists, it shares the same database as Dev so it is not a true standalone environment. 

Vision and Goals 

This section describes some of the shortcomings of the Jury v2 application currently being developed. The following items describe those areas and include steps to resolve the issues. 

Domain Implementation

While Jury is being developed using modern technologies, there is room for improvement in the way these technologies have been implemented.  In a rush to refresh the look and feel of the front-end, little planning was put into the internal architecture of the back end. As such the back end of the application does not conform to clean architecture principles. Currently the Application Layer (API) directly calls into the Persistence Layer (Database).  This implementation lacks a Domain Layer to centralize business logic into Domain Objects. The biggest negative impact of this design is code duplication. Instead of having a single class that implements logic to perform some task in the application, this logic can be implemented in several different places and in several different ways. Obviously this increases the complexity of implementing new features and increases the amount of code that must be tested. To address this problem, a single area of the application should be selected and rewritten to utilize domain objects in a domain layer. Once a few examples of this architecture have been created and demonstrated, this design pattern can be used to improve existing features and replicated when creating new features.

Containerization and Scaling

The Jury application consists of a front-end and back-end that can be deployed separately and thus lend themselves to containerization. Jury is currently deployed as two containers into a Kubernetes cluster in AWS. This gives us a great head start towards the ability to horizontally scale into multiple containerized instances to handle increased workloads. There is work that remains to be done, however, to ensure this works correctly. For example, tools such as Redis Cache or Amazon ElastiCache should be implemented so that cached data can be shared between instances of the container. In the future if smaller containers are needed, it would require minimal effort to move the BFF Authentication to its own container, and to split the API into separate containers. 

Security

There is a security framework in place inside the Jury back-end but this security framework is not yet being utilized. In the coming months this security framework must be implemented using the existing permission roles settings. This will allow locking down each API endpoint in the back-end and allow locking down each screen/menu option in the front-end. Examples of security implementation in the back-end and security implementation in the front-end must be done as examples for the developers. Once these examples are in place the functionality can be replicated throughout the application. 

Sensitive Data

Deployment of the Jury application is now being done using core scripts provided by DevOps. These scripts have since been updated by those working on the deployment of Jury to contain sensitive information such as database connection strings, S3 bucket secret keys, and Twilio secret keys, all in plain text.  Jury v2 must be enhanced to contain code to retrieve this information at runtime from a sensitive storage location such as AWS Secrets Manager. Once this code is in place, all sensitive information can be removed from the scripts and placed into the secrets store. DevOps can then manage access to this resource instead of storing this information in the deployment scripts. 

Automated Testing

Currently there is no automated testing occurring when the Jury application is built or deployed. There are examples of other products which use a combination of modern testing tools such as Playwright and SpecFlow to produce detailed reports about the source code quality. These testing technologies make use of natural languages such as Gherkin to create tests which match up to User Stories from the backlog. A good example of a Gherkin test case would be “When a new Summons is Drawn, a new Jury Pool should be Created".  Playwright utilizes SpecFlow to transform this natural language into a testing scenario and execute the test. Detailed artifacts are then created from the outcome of these tests and can be reviewed and acted upon immediately. 

Deployment

In addition to implementing automated testing inside the CI/CD pipeline, multiple environments and the ability to deploy directly to them must also be introduced. Existing are some scripts to promote deployments through the environments such as Dev to QA to Staging to Production. However, this process does not include the concept of frozen Releases or provide for the ability to release a version directly to an environment. For example, on March 1^st version 2.1.1 could be frozen and deployed to the QA environment. After some initial testing this version would then be promoted to Staging. In the meantime, a developer may have deployed a new feature that belongs in version 2.2.0 to the QA server for testing. If there is a bug found in version 2.1.1 of the Staging environment, that bug could be fixed, but there is no way to deploy version 2.1.2 back to the Staging environment. This bug fix would have to first be deployed to Dev then to QA then to Staging interrupting all work that is being done on the next version 2.2.0 in the QA environment. 

Architectural Principles 

Following are industry established architectural principles to be followed when designing new functionality for the application:  

• Modularity: Composed discrete, interchangeable modules with well-defined interfaces, promoting reusability, maintainability, and ease of understanding. 

• Scalability: Architect features to scale horizontally and vertically to accommodate growing user bases and increased workloads, leveraging distributed systems and efficient resource utilization. 

• Loose Coupling: Minimize dependencies between components to reduce the impact of changes and facilitate independent development, testing, and deployment of modules. 

• High Cohesion: Ensure that each module or component has a clear, focused purpose and functionality, promoting better maintainability, testability, and understanding of the codebase. 

• Separation of Concerns: Divide the application into distinct layers or components, each responsible for a specific aspect of functionality (e.g., presentation, business logic, data access), to improve modifiability and clarity of design. 

• Security by Design: Integrate security measures and best practices into the architecture from the outset, including authentication, authorization, encryption, and secure communication protocols, to protect against potential threats and vulnerabilities. 

• Performance Optimization: Architect the application to be performant by employing efficient algorithms, data structures, caching mechanisms, and optimization techniques to minimize latency, maximize throughput, and optimize resource utilization. 

• Flexibility and Extensibility: Design the application to accommodate future changes and enhancements by adopting flexible, extensible architectures and design patterns that allow for easy integration of new features, modules, and technologies. 

• Resilience and Fault Tolerance: Build the application to withstand and recover gracefully from failures, errors, and unexpected events by implementing fault-tolerant mechanisms, redundancy, graceful degradation, and automated recovery processes. 

• Testability: Ensure that the application is designed for ease of testing, with clear separation of concerns, modular architecture, and support for automated testing frameworks and tools to facilitate comprehensive testing and validation of functionality. 

• Simplicity and Minimalism: Strive for simplicity in design and implementation, favoring straightforward solutions over overly complex ones, to reduce cognitive overhead, minimize technical debt, and improve maintainability. 

• Adaptability to Change: Architect the application to be adaptable and responsive to changing business requirements, technological advancements, and user feedback, enabling iterative development, continuous integration, and delivery practices. 

Key Initiatives 

This section describes major architectural initiatives planned to minimize technical debt and address the shortcomings defined in the vision and goals section of the document. 

Initiative 1: Upgrade Back-end Framework 

• Description: The back-end of Jury v2 is currently .NET 6 / C# 10. We should target moving to .NET 8 LTS which will be supported through 11/10/2026. 

• Objective: Upgrade to .NET 8 to minimize technical debt. 

• Timeline: Q2 2024 

• Key Milestones:  

  • April Q2 2024: Update all Projects to .NET 8 / C# 12 

  • April Q2 2024: Identity features that are no longer supported in .NET 8 or that can be enhanced using the new language features of C# 12 

  • May Q2 2024: Incorporate deprecated features / C# enhancements into ongoing sprint work 

Initiative 2: Upgrade Front-end Framework 

• Description: Update the Angular framework to v17 as Angular v16 Security Support ends on 11/8/2024 

• Objective: Update Angular framework to v17 

• Timeline: Q2 2024 

• Key Milestones:  

  • April Q2 2024: Update Front-End to Angular v17 and perform initial testing 

  • May Q2 2024: Merge the update into the master branch 

  • May Q2 2024: May Deploy the updated master branch to the Dev environment 

  • May Q2 2024: Incorporate any additional issues identified into ongoing sprint work 

Initiative 3: Permissions Implementation 

• Description: While there is a framework within Jury v2 to implement permissions, every API endpoint and UI screen is accessible to all users. 

• Objective: Implement permissions on each API endpoint and each UI screen 

• Timeline: Q2 2024 - Q3 2024 

• Key Milestones:  

  • June Q2 2024: Implement an API endpoint with permissions 

  • June Q2 2024: Implement a UI screen / UI menu option with permissions 

  • June Q2 2024: Demonstrate and explain the implementation to the development team 

  • July Q3 2024: Incorporate implementation of all existing screens into ongoing sprint work 

Initiative 4:  Domain Implementation 

• Description: Jury v2 is missing a domain layer - not clean architecture  

• Objective: Build out Domain Layer to implement clean architecture 

• Timeline: Q4 2024 – Q2 2025 

• Key Milestones: 

  • March Q1 2024: Identity an existing Command / Query Handler which can be rearchitected to make use of a domain object 

  • April Q2 2024: Demonstrate and explain the implementation to the development team 

  • Q2 2024 - Q2 2025: Incorporate implementation of all existing screens into ongoing sprint work 

Initiative 5: Remove Sensitive Information 

• Description: Currently all sensitive information (database password, Twilio password, S3 password) is stored in plain text on the build server. The deployment process echoes the passwords to the screen making them visible even when simply checking to see if the deployment was successful. 

• Objective: Implement AWS Secrets Manager for all sensitive information 

• Timeline: Q4 2024 

• Key Milestones: 

  • October Q4 2024: Create a library within Jury v2 that can retrieve secrets from AWS Secrets Manager 

  • October Q4 2024: Use this library to retrieve sensitive information in one area of the application 

  • October Q4 2024: Demonstrate and explain the library during a presentation to the development team 

  • November Q4 2024: Incorporate implementation of the remaining secrets into ongoing sprint work 

Dependencies and Risks 

Identification of dependencies between architectural initiatives and potential risks (e.g., resource constraints, technology adoption challenges) that could impact successful implementation. 

• Shortage of resources and resource turnover may extend deliverables 

• Most development is done by reading old code and rewriting it into the new version. It is possible features are being implemented incorrectly due to the misunderstanding of old complex code. 

• Unfamiliarity with the AWS platform may cause some infrastructure tasks to drag (Secrets management, Deployment management, etc.) 

Resource Allocation 

1. Upgrade Back-end Framework 

   1. Resource Allocation: Entire Jury V2 team. 

   2. Training Needs: Review .NET and C# upgrade documentation. 

2. Upgrade Front-end Framework 

   1. Resource Allocation: Entire Jury V2 team. 

   2. Training Needs: None 

3. Permissions Implementation 

   1. Resource Allocation: Entire Jury V2 team, Richard as knowledgebase, Pete as Product resource, Bradey for integration with Identity Server. 

   2. Training Needs: Cross-training on prior permissions with Richard. 

4. Domain Implementation 

   1. Resource Allocation: Entire Jury V2 team, Bradey as knowledgebase. 

   2. Training Needs: General training on domain implementations. 

5. Remove Sensitive Information 

   1. Resource Allocation: 1-2 members of Jury V2 team. 

   2. Training Needs: Cross-training with other teams who have done the same changes. 

Governance and Oversight 

 Purpose 

This plan establishes a clear framework for software development and architecture governance within the Jury V2 project. This framework will guide the design, development, and maintenance of our systems, ensuring consistency, quality, and alignment with our business objectives. 

Scope 

This plan covers all aspects of software development and architecture for the Jury V2 product, including design principles, development processes, architectural standards, and governance mechanisms. 

Architecture Vision 

Strategic Goals 

• Enhance user experience and accessibility. 

• Ensure scalability and performance. 

• Maintain security and compliance standards. 

• Facilitate integration with external systems. 

Key Principles 

• Modularity: Design systems as a collection of independent, interchangeable modules. 

• Interoperability: Ensure systems can exchange and use information seamlessly. 

• Scalability: Design systems to handle increasing loads gracefully. 

• Security: Embed security considerations into every aspect of the architecture. 

Development Process 

Agile Methodology 

• Sprints: Develop features in short, iterative cycles (sprints).  Sprints will last 2 weeks. 

• Backlog Management: Maintain a prioritized list of user stories and tasks. 

• Continuous Integration/Continuous Deployment (CI/CD): Automate testing and deployment processes. 

Quality Assurance 

• Code Reviews: Conduct regular code reviews to ensure code quality and adherence to standards. 

• Automated Testing: Implement a comprehensive suite of automated tests (unit, integration, and end-to-end). 

• Performance Monitoring: Continuously monitor system performance and address any issues proactively. 

Architecture Governance 

Architecture Review Board (ARB) 

Establish an ARB comprising key stakeholders to oversee architectural decisions. The ARB will: 

• Review and approve architectural designs and changes. 

• Ensure alignment with strategic goals and compliance with standards. 

• Resolve architectural conflicts and issues. 

Standards and Guidelines 

Develop and maintain a set of architectural standards and guidelines, covering: 

• Coding standards 

• Security protocols 

• Data management practices 

• Integration patterns 

Compliance and Auditing 

Implement mechanisms to ensure compliance with architectural standards and guidelines, including: 

• Regular audits of code and architecture 

• Automated tools to enforce standards 

• Reporting and addressing non-compliance issues 

Communication Plan 

Meetings 

Architecture Project Kick-off Meeting: 

Frequency: At the project initiation phase. 

Attendees: All stakeholders. 

Purpose: Introduce the project, its goals, roles, and expectations. 

Bi-Weekly Status Meetings: 

Frequency: Every other [Specify Day] at [Specify Time]. 

Attendees: Software Development. 

Purpose: Discuss progress, address issues, and plan for the upcoming sprint.  Decide if changes are needed to architectural plan. 

Sprint Planning Meetings: 

Frequency: At the beginning of each sprint. 

Attendees: Software Development, Product Management, and Project Management teams. 

Purpose: Define sprint goals, prioritize tasks, and align on the scope. 

Sprint Review/Demo Meetings: 

Frequency: Every other [Specify Day] at [Specify Time]. 

Attendees: Software Development, Implementation, Product Management, and Project Management teams. 

Purpose: Demonstrate work that has been completed, discuss sprint goals and what was completed.  Architecture work will be part of normal sprint demo time. 

Communication Tools 

Roles and Responsibilities 

Escalation Process 

In case of unresolved issues or conflicts, the following escalation process will be followed: 

Reporting 

Regular status reports will be shared with all stakeholders, summarizing progress, issues, and action items. These reports will be distributed after each bi-weekly status meeting. 

Metrics and KPIs 

Scalability: Measure the system's ability to handle increased workload or growth in data volume, users, or transactions. 

Reliability: Evaluate the system's ability to perform consistently and predictably under varying conditions. 

Maintainability: Assess how easily the system can be maintained, updated, and enhanced over time. 

Interoperability: Evaluate the system's ability to work seamlessly with other systems or components. 

Compliance: Ensure that the system meets regulatory, legal, and contractual requirements. Also ensure the security ticket backlog is regularly mitigated. 

Review and Update Process 

To ensure that the architectural roadmap of the Jury V2 product is regularly reviewed and updated to align with evolving business needs, incorporate emerging technologies, and address feedback from stakeholders. 

1. Initiation: The review and update process is initiated quarterly, triggered by the end of each quarter. 

2. Stakeholder Identification: Identify key stakeholders who will contribute to the review and update process, including product managers, implementation personnel, developers, QA, architects, and any other relevant parties. 

3. Feedback Collection: 

   1. Stakeholders provide feedback on the current architectural roadmap, highlighting areas for improvement, emerging trends, and business needs. 

   2. Feedback can be collected through surveys, interviews, and workshops. 

4. Review Meeting: 

   1. A review meeting is scheduled with key stakeholders to discuss the feedback and assess the current architectural roadmap. 

   2. The meeting agenda includes a review of the feedback, identification of key areas for update, and prioritization of changes. 

5. Update Proposal: 

   1. Based on the review meeting, a proposal is drafted to update the architectural roadmap. 

   2. The proposal includes a summary of feedback, proposed changes to the roadmap, rationale for the changes, and estimated impact on the product. 

6. Approval Process: 

   1. The update proposal is presented to the ARB and development management for approval. 

   2. The ARB and development management team evaluates the proposal based on alignment with business goals, feasibility, and resource availability. 

7. Implementation Planning: 

   1. Once approved, an implementation plan is developed detailing the steps, timeline, and resources required to implement the updates. 

   2. The plan includes the previous communication plan to inform stakeholders about the upcoming changes. 

8. Implementation: 

   1. The updates to the architectural roadmap are implemented according to the plan. 

   2. Developers and architects collaborate to ensure that the updates are integrated smoothly into the development process. 

9. Monitoring and Feedback: 

   1. Post-implementation, the updated roadmap is monitored closely to assess its impact on the product. 

   2. Feedback is collected from stakeholders to evaluate the effectiveness of the updates and identify any further improvements needed. 

10. Documentation: 

    1. All updates to the architectural roadmap, including feedback, proposals, and implementation plans, are documented for future reference. 

    2. Documentation is made accessible to stakeholders for transparency and accountability. 

11. Continuous Improvement: 

    1. The review and update process is iterative, with regular reviews scheduled to ensure that the architectural roadmap remains aligned with business needs and technology trends. 

    2. Lessons learned from each review cycle are used to improve the process for future updates. 

Conclusion

The review and update process outlined above ensures that the architectural roadmap of the product is regularly reviewed and updated to meet the evolving needs of the business, incorporate new technologies, and address feedback from stakeholders. This process helps to ensure that the product remains competitive, innovative, and aligned with the strategic goals of the organization.