Jury v2 Architectural Roadmap
- 1 Executive Summary
- 2 Current Architecture
- 3 Vision and Goals
- 4 Architectural Principles
- 5 Key Initiatives
- 6 Dependencies and Risks
- 7 Resource Allocation
- 8 Governance and Oversight
- 9 Architecture Vision
- 9.1 Strategic Goals
- 9.2 Key Principles
- 10 Development Process
- 11 Architecture Governance
- 12 Communication Plan
- 13 Metrics and KPIs
- 14 Review and Update Process
- 15 Conclusion
Executive Summary
The Jury v2 software architectural roadmap focuses on modernizing the technology stack, enhancing performance and scalability, and improving security. The current architecture utilizes Angular v16, .NET 6, and various other technologies, deployed in a containerized environment on AWS. Key initiatives include upgrading to Angular v17, implementing permissions, building out a domain layer, and securing sensitive information with AWS Secrets Manager. The roadmap aims to address technical debt, improve system reliability, and ensure compliance with security standards.
Current Architecture
The information below details the technologies used in Jury v2 and describes the environment in which Jury v2 is deployed.
Technology Stack
Front-end:
Angular v16: UI framework developed by Google
Angular Material v16: Prebuilt UI Components developed for use in Angular
Tailwind CSS v3.4.1: Library used to style HTML elements
Authentication:
OpenID Connect (OIDC): industry standard authentication protocol
JSON Web Tokens (JWT): industry standard security token format
Back-end For Front-end (BFF) Encryption Layer: secure client token management library developed by Duende Software
Microsoft ASP.NET Core authentication libraries: latest encryption/decryption/verification libraries provided by Microsoft
Back-end:
Microsoft .NET v6 with C# v10
Command Query Responsibility Segregation (CQRS): Separates the concerns of the Command (create, update, and delete) operations from the Query(read) operations
Dapper: An object relational mapping (ORM) library which simplifies the create, read, update, and delete (CRUD) operations on the database by automatically mapping database parameters and data results to C# objects
Fluent Validation: A library which allows developers to easily implement complex validation rules on all data sent to the API
MediatR: an in-memory bus which implements a publisher/subscriber (Pub/Sub) used to loosely couple the API layer to the Application Layer
NodaTime: A date time management library used to extend the limited date time functionality provided by the .NET framework. Usage of frameworks such as this are key when implementing multi-tenant solutions as tenants and users are dispersed throughout the world and rarely in the same time zone as the servers.
QuestPDF: This is a newly developed open-source library which is used to generate PDF documents using C#. A few reports were built using this technology, but we may phase them out and implement more current versions of the reporting tools (Aspose and ActiveReports) used in Jury v1.
Aspose: Reporting / Document generation library used to build PDF documents from templates which can be defined by users
ActiveReports: A reporting solution used to generate listings or statistical reports
Twilio: This library enables the application to send SMS Messages to Jurors
Authentication:
OpenID Connect (OIDC) - industry standard authentication protocol
JSON Web Tokens (JWT) - industry standard security token format
Microsoft ASP.NET Core authentication libraries – latest encryption/decryption/verification libraries provided by Microsoft
Database: Microsoft SQL Server
Document Storage: Amazon S3
Deployment Architecture
When deployed, the application has the following AWS Footprint:
3 Kubernetes Containers
Front-end:
OS: Debian Linux v11
Server: Microsoft Kestrel Web Server
Frameworks: ASP.NET Core 6.0
Utilization: Always On
Back-end:
OS: Debian Linux v11
Server: Microsoft Kestrel Web Server
Frameworks: ASP.NET Core 6.0
Utilization: Always On
Database Updater:
OS: Debian Linux v11
Application: .NET Core 7.0 Console Application
Frameworks: .NET SDK 7.0
Utilization: Deployment Only
1 AWS RDS Server running SQL Server Engine
1 Master Database
1 Database per Tenant
1 S3 Bucket per Tenant for Document Storage
Assessment of system scalability, performance, and reliability
Scalability: Although the application is deployed in a containerized manner, preliminary work has not yet been done to ensure the application can run in a scaled environment (multiple containers). Further research is needed to determine if we may need to implement a framework such as Redis or Amazon ElastiCache to share information between servers.
Database Performance: The database structure from Jury v1 has remained largely unchanged through Jury v2. As such, all performance improvements in the v1 database are contained in the v2 database. In addition, the database auditing mechanism utilized in Jury v1 was replaced in Jury v2 which removed over 300 complex triggers.
Application Performance: Jury v1 and Jury v2 are both built upon the .NET framework and all algorithms present in the Jury v1 application were ported to the Jury v2 application
Database Reliability: Failover Groups will have to be implemented on the database implementation to provide for additional database servers to spool up should the servers hosting the databases fail.
Application Reliability: As mentioned above in the scalability section, research is still needed into replicating containerized instances of the application. This work will yield information on what enhancements are required to seamlessly float containers across AWS Availability Zones in case of outages.
Identification of architectural bottlenecks and pain points
While Jury v2 does follow the CQRS pattern, there is no Domain Layer. Business Logic which should be contained within Domain Objects are spread throughout various Command Handlers. This causes duplication of code and is not considered clean architecture.
A framework to implement permissions exists in the backend however no code utilizes this framework. This means all API endpoints and all UI screens and features can be accessed by all users regardless of their configured permissions.
Evaluation of Security Posture
User authentication in Jury v2 was originally handled by a custom ASP.NET Identity solution implemented within the front-end and back-ends of the application. This custom solution was removed from Jury v2 and replaced with a much more secure integration to Duende Identity Server. This change allowed us to rip out thousands of lines of complex authorization/authentication code from both the front-end and the back-end.
Front-End security: Each request for a Web Page must contain an encrypted User Access Token which is decrypted and validated by the BFF layer before being allowed to pass to the Web Server.
Back-End security: Each request to an API Endpoint must contain an encrypted User Access Token which is decrypted and validated by the BFF layer before being allowed to pass to the API Server.
Jury v2 is regularly monitored by SonarQube and any code security issues are addressed as they arise.
Jury v2 will soon also be monitored by Qualys and any security vulnerabilities will be addressed as they arise
Database security: Currently the Jury v2 application uses a connection string with a user account containing a username and password to access the database. This should be updated to remove the user account from the connection string. Instead, the API application should be running under an AWS service account and this service account should be given access to the database. This would prevent someone from using the username and password found in the connection string to access the database.
Review of development and deployment processes
Development Process:
Each development task is performed on a fork of the master branch
A Pull Request (PR) is created by the developer for review by a senior team member
The PR is merged into the master branch
Development Process Issues:
Due to a lack of deployment environments, work cannot be verified by a QA or UX resource without first checking the code into the master branch. This results in potentially broken functionality being placed into the master branch.
Due to a shortage of resources assigned to the project, Pull Requests require only 1 reviewer instead of the standard 2 reviewers.
Deployment Process:
Deployment is limited as it currently only deploys to a single environment named Dev.
Deployment Process Issues:
Multiple deployment pipelines should be created so different features of the application can be deployed to different environments such as Test1, Test2, and Test3, etc. In addition, Release Candidate versions of the application should be deployed to Preview or Staging environments and analyzed before being pushed to Production. While there is a QA environment that exists, it shares the same database as Dev so it is not a true standalone environment.
Vision and Goals
This section describes some of the shortcomings of the Jury v2 application currently being developed. The following items describe those areas and include steps to resolve the issues.
Domain Implementation
While Jury is being developed using modern technologies, there is room for improvement in the way these technologies have been implemented. In a rush to refresh the look and feel of the front-end, little planning was put into the internal architecture of the back end. As such the back end of the application does not conform to clean architecture principles. Currently the Application Layer (API) directly calls into the Persistence Layer (Database). This implementation lacks a Domain Layer to centralize business logic into Domain Objects. The biggest negative impact of this design is code duplication. Instead of having a single class that implements logic to perform some task in the application, this logic can be implemented in several different places and in several different ways. Obviously this increases the complexity of implementing new features and increases the amount of code that must be tested. To address this problem, a single area of the application should be selected and rewritten to utilize domain objects in a domain layer. Once a few examples of this architecture have been created and demonstrated, this design pattern can be used to improve existing features and replicated when creating new features.
Containerization and Scaling
The Jury application consists of a front-end and back-end that can be deployed separately and thus lend themselves to containerization. Jury is currently deployed as two containers into a Kubernetes cluster in AWS. This gives us a great head start towards the ability to horizontally scale into multiple containerized instances to handle increased workloads. There is work that remains to be done, however, to ensure this works correctly. For example, tools such as Redis Cache or Amazon ElastiCache should be implemented so that cached data can be shared between instances of the container. In the future if smaller containers are needed, it would require minimal effort to move the BFF Authentication to its own container, and to split the API into separate containers.
Security
There is a security framework in place inside the Jury back-end but this security framework is not yet being utilized. In the coming months this security framework must be implemented using the existing permission roles settings. This will allow locking down each API endpoint in the back-end and allow locking down each screen/menu option in the front-end. Examples of security implementation in the back-end and security implementation in the front-end must be done as examples for the developers. Once these examples are in place the functionality can be replicated throughout the application.
Sensitive Data
Deployment of the Jury application is now being done using core scripts provided by DevOps. These scripts have since been updated by those working on the deployment of Jury to contain sensitive information such as database connection strings, S3 bucket secret keys, and Twilio secret keys, all in plain text. Jury v2 must be enhanced to contain code to retrieve this information at runtime from a sensitive storage location such as AWS Secrets Manager. Once this code is in place, all sensitive information can be removed from the scripts and placed into the secrets store. DevOps can then manage access to this resource instead of storing this information in the deployment scripts.
Automated Testing
Currently there is no automated testing occurring when the Jury application is built or deployed. There are examples of other products which use a combination of modern testing tools such as Playwright and SpecFlow to produce detailed reports about the source code quality. These testing technologies make use of natural languages such as Gherkin to create tests which match up to User Stories from the backlog. A good example of a Gherkin test case would be “When a new Summons is Drawn, a new Jury Pool should be Created". Playwright utilizes SpecFlow to transform this natural language into a testing scenario and execute the test. Detailed artifacts are then created from the outcome of these tests and can be reviewed and acted upon immediately.
Deployment
In addition to implementing automated testing inside the CI/CD pipeline, multiple environments and the ability to deploy directly to them must also be introduced. Existing are some scripts to promote deployments through the environments such as Dev to QA to Staging to Production. However, this process does not include the concept of frozen Releases or provide for the ability to release a version directly to an environment. For example, on March 1st version 2.1.1 could be frozen and deployed to the QA environment. After some initial testing this version would then be promoted to Staging. In the meantime, a developer may have deployed a new feature that belongs in version 2.2.0 to the QA server for testing. If there is a bug found in version 2.1.1 of the Staging environment, that bug could be fixed, but there is no way to deploy version 2.1.2 back to the Staging environment. This bug fix would have to first be deployed to Dev then to QA then to Staging interrupting all work that is being done on the next version 2.2.0 in the QA environment.
Architectural Principles
Following are industry established architectural principles to be followed when designing new functionality for the application:
Modularity: Composed discrete, interchangeable modules with well-defined interfaces, promoting reusability, maintainability, and ease of understanding.
Scalability: Architect features to scale horizontally and vertically to accommodate growing user bases and increased workloads, leveraging distributed systems and efficient resource utilization.
Loose Coupling: Minimize dependencies between components to reduce the impact of changes and facilitate independent development, testing, and deployment of modules.
High Cohesion: Ensure that each module or component has a clear, focused purpose and functionality, promoting better maintainability, testability, and understanding of the codebase.
Separation of Concerns: Divide the application into distinct layers or components, each responsible for a specific aspect of functionality (e.g., presentation, business logic, data access), to improve modifiability and clarity of design.
Security by Design: Integrate security measures and best practices into the architecture from the outset, including authentication, authorization, encryption, and secure communication protocols, to protect against potential threats and vulnerabilities.
Performance Optimization: Architect the application to be performant by employing efficient algorithms, data structures, caching mechanisms, and optimization techniques to minimize latency, maximize throughput, and optimize resource utilization.
Flexibility and Extensibility: Design the application to accommodate future changes and enhancements by adopting flexible, extensible architectures and design patterns that allow for easy integration of new features, modules, and technologies.
Resilience and Fault Tolerance: Build the application to withstand and recover gracefully from failures, errors, and unexpected events by implementing fault-tolerant mechanisms, redundancy, graceful degradation, and automated recovery processes.
Testability: Ensure that the application is designed for ease of testing, with clear separation of concerns, modular architecture, and support for automated testing frameworks and tools to facilitate comprehensive testing and validation of functionality.
Simplicity and Minimalism: Strive for simplicity in design and implementation, favoring straightforward solutions over overly complex ones, to reduce cognitive overhead, minimize technical debt, and improve maintainability.
Adaptability to Change: Architect the application to be adaptable and responsive to changing business requirements, technological advancements, and user feedback, enabling iterative development, continuous integration, and delivery practices.
Key Initiatives
This section describes major architectural initiatives planned to minimize technical debt and address the shortcomings defined in the vision and goals section of the document.
Initiative 1: Upgrade Back-end Framework
Description: The back-end of Jury v2 is currently .NET 6 / C# 10. We should target moving to .NET 8 LTS which will be supported through 11/10/2026.
Objective: Upgrade to .NET 8 to minimize technical debt.
Timeline: Q2 2024
Key Milestones:
April Q2 2024: Update all Projects to .NET 8 / C# 12
April Q2 2024: Identity features that are no longer supported in .NET 8 or that can be enhanced using the new language features of C# 12
May Q2 2024: Incorporate deprecated features / C# enhancements into ongoing sprint work
Initiative 2: Upgrade Front-end Framework
Description: Update the Angular framework to v17 as Angular v16 Security Support ends on 11/8/2024
Objective: Update Angular framework to v17
Timeline: Q2 2024
Key Milestones:
April Q2 2024: Update Front-End to Angular v17 and perform initial testing
May Q2 2024: Merge the update into the master branch
May Q2 2024: May Deploy the updated master branch to the Dev environment
May Q2 2024: Incorporate any additional issues identified into ongoing sprint work
Initiative 3: Permissions Implementation
Description: While there is a framework within Jury v2 to implement permissions, every API endpoint and UI screen is accessible to all users.
Objective: Implement permissions on each API endpoint and each UI screen
Timeline: Q2 2024 - Q3 2024
Key Milestones:
June Q2 2024: Implement an API endpoint with permissions
June Q2 2024: Implement a UI screen / UI menu option with permissions
June Q2 2024: Demonstrate and explain the implementation to the development team
July Q3 2024: Incorporate implementation of all existing screens into ongoing sprint work
Initiative 4: Domain Implementation
Description: Jury v2 is missing a domain layer - not clean architecture
Objective: Build out Domain Layer to implement clean architecture
Timeline: Q4 2024 – Q2 2025
Key Milestones:
March Q1 2024: Identity an existing Command / Query Handler which can be rearchitected to make use of a domain object
April Q2 2024: Demonstrate and explain the implementation to the development team
Q2 2024 - Q2 2025: Incorporate implementation of all existing screens into ongoing sprint work
Initiative 5: Remove Sensitive Information
Description: Currently all sensitive information (database password, Twilio password, S3 password) is stored in plain text on the build server. The deployment process echoes the passwords to the screen making them visible even when simply checking to see if the deployment was successful.
Objective: Implement AWS Secrets Manager for all sensitive information
Timeline: Q4 2024
Key Milestones:
October Q4 2024: Create a library within Jury v2 that can retrieve secrets from AWS Secrets Manager
October Q4 2024: Use this library to retrieve sensitive information in one area of the application
October Q4 2024: Demonstrate and explain the library during a presentation to the development team
November Q4 2024: Incorporate implementation of the remaining secrets into ongoing sprint work
Dependencies and Risks
Identification of dependencies between architectural initiatives and potential risks (e.g., resource constraints, technology adoption challenges) that could impact successful implementation.
Shortage of resources and resource turnover may extend deliverables
Most development is done by reading old code and rewriting it into the new version. It is possible features are being implemented incorrectly due to the misunderstanding of old complex code.
Unfamiliarity with the AWS platform may cause some infrastructure tasks to drag (Secrets management, Deployment management, etc.)
Resource Allocation
Upgrade Back-end Framework
Resource Allocation: Entire Jury V2 team.
Training Needs: Review .NET and C# upgrade documentation.
Upgrade Front-end Framework
Resource Allocation: Entire Jury V2 team.
Training Needs: None
Permissions Implementation
Resource Allocation: Entire Jury V2 team, Richard as knowledgebase, Pete as Product resource, Bradey for integration with Identity Server.
Training Needs: Cross-training on prior permissions with Richard.
Domain Implementation
Resource Allocation: Entire Jury V2 team, Bradey as knowledgebase.
Training Needs: General training on domain implementations.
Remove Sensitive Information
Resource Allocation: 1-2 members of Jury V2 team.
Training Needs: Cross-training with other teams who have done the same changes.
Governance and Oversight
Purpose
This plan establishes a clear framework for software development and architecture governance within the Jury V2 project. This framework will guide the design, development, and maintenance of our systems, ensuring consistency, quality, and alignment with our business objectives.
Scope
This plan covers all aspects of software development and architecture for the Jury V2 product, including design principles, development processes, architectural standards, and governance mechanisms.
Architecture Vision
Strategic Goals
Enhance user experience and accessibility.
Ensure scalability and performance.
Maintain security and compliance standards.
Facilitate integration with external systems.
Key Principles
Modularity: Design systems as a collection of independent, interchangeable modules.
Interoperability: Ensure systems can exchange and use information seamlessly.
Scalability: Design systems to handle increasing loads gracefully.
Security: Embed security considerations into every aspect of the architecture.
Development Process
Agile Methodology
Sprints: Develop features in short, iterative cycles (sprints). Sprints will last 2 weeks.
Backlog Management: Maintain a prioritized list of user stories and tasks.
Continuous Integration/Continuous Deployment (CI/CD): Automate testing and deployment processes.
Quality Assurance
Code Reviews: Conduct regular code reviews to ensure code quality and adherence to standards.
Automated Testing: Implement a comprehensive suite of automated tests (unit, integration, and end-to-end).
Performance Monitoring: Continuously monitor system performance and address any issues proactively.
Architecture Governance
Architecture Review Board (ARB)
Establish an ARB comprising key stakeholders to oversee architectural decisions. The ARB will:
Review and approve architectural designs and changes.
Ensure alignment with strategic goals and compliance with standards.
Resolve architectural conflicts and issues.
Standards and Guidelines
Develop and maintain a set of architectural standards and guidelines, covering:
Coding standards
Security protocols
Data management practices
Integration patterns
Compliance and Auditing
Implement mechanisms to ensure compliance with architectural standards and guidelines, including:
Regular audits of code and architecture
Automated tools to enforce standards
Reporting and addressing non-compliance issues
Communication Plan
Meetings
Architecture Project Kick-off Meeting:
Frequency: At the project initiation phase.
Attendees: All stakeholders.
Purpose: Introduce the project, its goals, roles, and expectations.
Bi-Weekly Status Meetings:
Frequency: Every other [Specify Day] at [Specify Time].
Attendees: Software Development.
Purpose: Discuss progress, address issues, and plan for the upcoming sprint. Decide if changes are needed to architectural plan.
Sprint Planning Meetings:
Frequency: At the beginning of each sprint.
Attendees: Software Development, Product Management, and Project Management teams.
Purpose: Define sprint goals, prioritize tasks, and align on the scope.
Sprint Review/Demo Meetings:
Frequency: Every other [Specify Day] at [Specify Time].
Attendees: Software Development, Implementation, Product Management, and Project Management teams.
Purpose: Demonstrate work that has been completed, discuss sprint goals and what was completed. Architecture work will be part of normal sprint demo time.
Communication Tools
Email: Use for formal announcements, reports, and documentation sharing.
Project Management Tool (JIRA): Use for task tracking, updates, and collaboration. This is also the Dev tool for tracking work.
Instant Messaging (Microsoft Teams): Use for quick updates, queries, and informal discussions.
Video Conferencing (Microsoft Teams): Use for remote meetings, workshops, and brainstorming sessions.
Roles and Responsibilities
Team Architect: Chosen member(s) from the team that will act as the architect for the software product and drive architecture work forward. They will assume responsibility for architecture plan movement in lieu of a project manager.
Team Leads (Development and Implementation): Responsible for team-specific communication and reporting.
Product Manager: Responsible for product-related communication, prioritizing features, and gathering feedback. In lieu of a Product Manager, the team will reach out to Project Managers for Product-specific guidance.
Escalation Process
In case of unresolved issues or conflicts, the following escalation process will be followed:
Level 1: Team Leads will attempt to resolve issues within their teams.
Level 2: If the issue remains unresolved, it will be escalated to the Project Manager.
Level 3: If necessary, the issue will be further escalated to the respective department heads or higher management.
Reporting
Regular status reports will be shared with all stakeholders, summarizing progress, issues, and action items. These reports will be distributed after each bi-weekly status meeting.
Metrics and KPIs
Scalability: Measure the system's ability to handle increased workload or growth in data volume, users, or transactions.
KPIs: Response time under load; resource utilization.
Reliability: Evaluate the system's ability to perform consistently and predictably under varying conditions.
KPIs: Mean time between failures; mean time to recover; error rate.
Maintainability: Assess how easily the system can be maintained, updated, and enhanced over time.
KPIs: Code complexity (measured by SQ); time to develop and deploy new features.
Interoperability: Evaluate the system's ability to work seamlessly with other systems or components.
KPIs: Time to successful integration acceptance; failure/success rate of integration tests.
Compliance: Ensure that the system meets regulatory, legal, and contractual requirements. Also ensure the security ticket backlog is regularly mitigated.
KPIs: SOC2 audit compliance results; customer acceptance of regulatory changes; number of reported incidents; security backlog and scanning through SonarQube and Qualys.
Review and Update Process
To ensure that the architectural roadmap of the Jury V2 product is regularly reviewed and updated to align with evolving business needs, incorporate emerging technologies, and address feedback from stakeholders.
Initiation: The review and update process is initiated quarterly, triggered by the end of each quarter.
Stakeholder Identification: Identify key stakeholders who will contribute to the review and update process, including product managers, implementation personnel, developers, QA, architects, and any other relevant parties.
Feedback Collection:
Stakeholders provide feedback on the current architectural roadmap, highlighting areas for improvement, emerging trends, and business needs.
Feedback can be collected through surveys, interviews, and workshops.
Review Meeting:
A review meeting is scheduled with key stakeholders to discuss the feedback and assess the current architectural roadmap.
The meeting agenda includes a review of the feedback, identification of key areas for update, and prioritization of changes.
Update Proposal:
Based on the review meeting, a proposal is drafted to update the architectural roadmap.
The proposal includes a summary of feedback, proposed changes to the roadmap, rationale for the changes, and estimated impact on the product.
Approval Process:
The update proposal is presented to the ARB and development management for approval.
The ARB and development management team evaluates the proposal based on alignment with business goals, feasibility, and resource availability.
Implementation Planning:
Once approved, an implementation plan is developed detailing the steps, timeline, and resources required to implement the updates.
The plan includes the previous communication plan to inform stakeholders about the upcoming changes.
Implementation:
The updates to the architectural roadmap are implemented according to the plan.
Developers and architects collaborate to ensure that the updates are integrated smoothly into the development process.
Monitoring and Feedback:
Post-implementation, the updated roadmap is monitored closely to assess its impact on the product.
Feedback is collected from stakeholders to evaluate the effectiveness of the updates and identify any further improvements needed.
Documentation:
All updates to the architectural roadmap, including feedback, proposals, and implementation plans, are documented for future reference.
Documentation is made accessible to stakeholders for transparency and accountability.
Continuous Improvement:
The review and update process is iterative, with regular reviews scheduled to ensure that the architectural roadmap remains aligned with business needs and technology trends.
Lessons learned from each review cycle are used to improve the process for future updates.
Conclusion
The review and update process outlined above ensures that the architectural roadmap of the product is regularly reviewed and updated to meet the evolving needs of the business, incorporate new technologies, and address feedback from stakeholders. This process helps to ensure that the product remains competitive, innovative, and aligned with the strategic goals of the organization.