In today's fast-paced tech world, embracing Generative AI (GenAI) has shifted from being a trend to becoming an essential element for organizations looking to optimize operations and enhance efficiency. GenAI tools offer immense potential to revolutionize daily workflows, streamline processes, and improve productivity. However, there's no universal roadmap for implementing these tools, making it crucial for companies to explore and experiment to find what works best.
At VGS, we formed a dedicated GenAI workgroup. Our mission was to investigate how AI could meet our immediate needs while planning for our journey to evolve as AI technology advances. Let's dive into the key areas the workgroup has focused on and the valuable insights they've gained along the way.
GenAI for R&D efficiency
The VGS GenAI workgroup honed in on three critical areas to improve R&D efficiency and boost productivity:
- GenAI tools for daily engineering tasks
- GPT models for operational assistance
- LLMs with RAG (Retrieval-Augmented Generation) support for fine-tuning information retrieval and references
During our Gen AI tools evaluation, the team encountered some challenges:
- Lack of clear policies around data privacy and indemnification
- Tools in their early development stage being better suited for early-stage partnerships.
- Repositories readiness for GenAI tools usage (i.e. removal of any secrets in the repo)
Nevertheless, the workgroup persevered.
Key Areas of Focus
The focus on enhancing R&D efficiency allowed the team to sift through the wide array of GenAI tools available. We zeroed in on those that directly aligned with our goals and selected that would have the most significant impact. The primary areas targeted for improvement were:
- R&D infrastructure
- Application development
- Support for security audits
Establishing a GenAI Policy
A key step in VGS's GenAI journey was creating a Generative AI policy as the study began. This policy wasn't just about selecting the right tools; it was also about ensuring strict compliance with data privacy and indemnification requirements. Collaborating with the legal and security teams, the workgroup drafted an External Generative AI policy based on the FS-ISAC Framework for Acceptable Use of Generative AI.
This policy helped define:
- The boundaries of data usage
- Standards for security compliance
- Acceptable practices for data privacy and indemnification
This policy setting made it easier to narrow down the list of GenAI tools to only those that met VGS's rigorous security requirements.
Choosing the Right Subscription Models
When recommending GenAI tools or services, the workgroup didn't stop at functionality. We also offered guidance on selecting the right subscription models, ensuring that the tools chosen were both practical and cost-effective.
For example, we:
- Identified team members who specifically needed GenAI assistance for code development
- Assessed how many tokens were required for RAG usage
- Identified the level of GPT subscription based on projected usage
Addressing these questions in advance helped the workgroup to prevent unexpected costs, keep recurring expenses in check and ensure the tools delivered real value.
A Slow and Steady Rollout
Integrating new tools into workflows is never an overnight process, and the integration was more critical with new GenAI tools. To ease this transition, the GenAI workgroup shared best practices with the R&D team, including:
- Prompt engineering with examples of effective prompts
- Procedures for generating GPT-powered bots
Our approach aimed to accelerate adoption of these tools, enabling team members to incorporate them into their daily routines seamlessly.
New Comprehensive Policies for Data Security
In addition to developing best practices in collaboration with the R&D team for the implementation of GenAI tools, the workgroup has established comprehensive policies concerning repository sanitization. These policies stipulate that repositories must be devoid of any sensitive data, including secrets, environmental details, configuration settings, and customer data (including test data).
Any team seeking to adopt GenAI tools is required to generate a sanitized report using a security scanning tool for the relevant repository. This report must be submitted for approval prior to the deployment of the GenAI tools.
Our approach significantly mitigates the risk of data leaks or the inadvertent use of sensitive data in model training.
Evaluating GenAI Tools for Code Assistance and Chatbots
The workgroup's approach went beyond ensuring compliance with data privacy, confidentiality, and indemnification standards. In order to select the right GenAI tools, we developed a set of specific criteria to evaluate each tool's effectiveness.
RESULTS
Based on our studies and evaluations, the GenAI workgroup selected:
- Github Copilot as our GenAI development assistant
- OpenAI ChatGPT as our workplace assistant Chatbot.
For LLMs with RAG support, we continue to explore our options to select tools that align with VGS's unique requirements.
Deployment and Evaluation on Effectiveness
To ensure these tools deliver real value, VGS R&D is planning a multi-step approach:
- Targeted deployments, including training sessions on best practices and effective prompting techniques to help engineers quickly start benefiting from GenAI tools in their daily development work. We were not adamant on perfection, but rather encouraged building familiarity with experimentation.
- Metrics tracking to evaluate tool effectiveness. The Engineering Enablement team will track and monitor key performance indicators such as tool usage frequency, pull request generation and merge rates, code coverage, and build success rates. These insights will help determine whether onboarding processes need refinement or if tool selection should be adjusted.
VGS will ensure that GenAI tools enhance productivity and contribute to the overall quality of development processes by consistently monitoring these metrics.
Looking Ahead
As the GenAI landscape rapidly evolves, VGS is committed to staying at the forefront of innovation. While the current focus is on R&D, the potential for GenAI tools to benefit other departments within the organization is immense. As adoption expands, the workgroup plans to continuously reevaluate the tools and services available, identifying new opportunities to enhance operational efficiency and reduce costs across the board.
While GenAI technology will undoubtedly continue to evolve, the proactive approach taken by the VGS GenAI workgroup positions the company to remain ahead of the curve. This is just the beginning of an exciting journey, and the possibilities for leveraging AI are truly limitless.
