Established an AI validation framework using PromptFoo and Azure AI Foundry.
Defined evaluation suites covering prompt regressions, tool-use correctness, hallucination rates, and high-level (e2e) success rate. Added into CI pipelines and evaluated and gathered telemetry over time.
Azure AI Foundry was used for model hosting, dataset management and tracing, and PromptFoo for the assertions and grading.
Built an autonomous exploratory testing agent using PydanticAI on top of a Databricks-hosted model and data platform.
The agent reasons about application state, makes decisions based on code deltas, plans exploration paths, creates invariants, then drives the system under test and finds issues. It augments daily workflows and speeds up the development process.
Databricks was used for model serving and agent creation. PydanticAI simplified the implementation and provided many tools out of the box.
Architected, deployed, managed and tested a solution for running Playwright tests in Azure in a scalable manner, using a custom GUI.
Basic architecture consisted of: App Service, Storage Queues, Azure Container Apps, App Gateway and Application Insights.
Used technologies such as Teraform, React, FastAPI, Dotnet, WebSockets, Electron, k6, Docker.
Comprehensive testing was done using such methods as unit, component, integration, end-to-end and property-based testing.
Swarming large numbers of clients for stress testing was made possible using Playwright scripts.
Implemented contract testing with Pact across a Kubernetes-based microservice architecture, as well as fault-injection testing.
Pact brokers were integrated into CI/CD pipelines, which has prevented many API misalignments in the five-team project.
Fault-injection scenarios, such as increased latency, dropped packets, pod kills, dependency timeouts, were introduced to confirm graceful degradation, retry behaviour and circuit-breaker pattern under realistic failure modes.
Workflows and processes around Grafana k6 Cloud and Grafana k6 were designed and put in place.
Custom tools and extensions for various protocols, company-wide integrations and customizations were developed. Development teams were taught, onboarded and supported.
Tailored observability dashboarding services and synthetic monitoring were implemented.
A Selenium-based framework with integrations with a plethora of tools used across the company was implemented, enhanced and supported.
Client requests for features accommodated and built in.
Ongoing support in terms of onboarding, learning and technical difficulties was provided.
Created a custom dataset based on the internal (in-house products) and vendor-provided documentation. Created multiple hot-swappable rank-16 adapters for an open-weight model using 4-bit NF4 QLoRA. Deployed the models using Databricks and validated their correctness using DeepEval. Deployed models were used for internal purposes, such as code generation (company specific frameworks), answering questions, debugging and tool use.
The data was augmented with synthetic Q/A pairs distilled from a frontier teacher model and manually filtered. G-Eval, hallucination, Pass@k metrics were used for validation.
Designed and built a desktop test automation recording solution combining Playwright, NutJS and an in-house AI integration.
Playwright test runner provided the execution layer, NutJS desktop interactions, and the AI layer translated recorded user sessions into test scripts.
The result was a recorder capable of capturing end-to-end flows that span browser and desktop - and producing maintainable and deterministic test code.
Designed a CI/CD/CT solution for dynamic AEM content deployment to GCP, together with an extensive testing suite (performance, visual, functional, accessibility, Web Vitals, component, unit). The framework allowed the design and creation teams to securily and easily deploy changes to production environment in a timely fashion, preventing defects and issues from being published in an interactive manner.
Designed a data testing solution that applies Great Expectations to streaming data flowing through Kafka topics.
Expectation suites were authored for each critical stream, validating schema, distributions, referential integrity and freshness - failures triggered alerting and quarantine routing rather than silently polluting downstream systems.
The integration sampled topics continuously, materialised validation runs and surfaced data quality trends, giving data consumers the same level of confidence in real-time pipelines that traditional test suites provide for code.
Comprehensive CI/CD solutions for multiple complex web platforms were created using Cypress, Vitest, visual testing and Github Actions.
Multilayered automated quality gates were put in place to secure the delivery pipeline and prevent defects from progressing to production.
The use of model-based testing with automatic shrinking combined with visual testing allowed to increase the confidence in the product and drastically shortened the delivery time necessary.
Designed and deployed an online raport generator with data aggregation.
Data from automated tests using Selenium, Cypress, Playwright, Lighthouse, Axe and Vitest were correlated and raports in a specific format were generated and stored.
Technologies such as Ligthouse Server, Azure CosmosDB, Azure Functions were used.