Skip to content

DO-2K23-26/mlops-bif-v2

Repository files navigation

MLOps project

DVC : Use a remote storage to store data

dvc init
dvc add data/raw/dataset.csv
dvc remote add -d s3-storage s3://mlops-bif
dvc remote modify s3-storage endpointurl https://b09595e7413d52541d55ebc0bf445b9c.r2.cloudflarestorage.com/mlops-bif
dvc remote modify --local s3-storage access_key_id 'YOUR_S3_KEY'
dvc remote modify --local s3-storage secret_access_key 'YOUR_S3_SECRET_KEY'
dvc push

Mise : Use differents Python environment

For each projects:

mise trust
mise install
uv pip install '.[dev]'

Install the Model Registry

Do it once, at the beginning of the project.

First we create the model registry inside our namespace (csgo3)

kubectl apply -k .

to check when everything is running (it can take a few minutes)

kubectl get pods -n csgo3 | grep model-registry

we add a rule to bypass the istio policy :

cat << 'EOF' | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: model-registry-allow
  namespace: csgo3
spec:
  selector:
    matchLabels:
      component: model-registry-server
  action: ALLOW
  rules:
  - {}
EOF

then we dclare the model to the kubeflow registry

cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: model-registry-store
  namespace: kubeflow
  labels:
    app: model-registry-store
data:
  model-registry-service.csgo3: |
    {
      "displayName": "model-registry-service",
      "description": "CS:GO model registry",
      "apiAddress": "http://model-registry-service.csgo3.svc.cluster.local:8080",
      "externalAPIAddress": "",
      "modelRegistryNamespace": "csgo3"
    }
EOF

Environment variables

Create a .env file in the root of the project from the .env.template file and fill in the values.

Usage

Python scripts

To generate the pipeline:

uv pip install '.[dev]'
uv run compile

Kubernetes secrets

To create a Kubernetes secret from the .env file:

kubectl create secret generic dvc-credentials -n csgo3 --from-env-file=.env

About

Giada, Thomas and Baptiste's MLOps project V2 (from Alexandre project)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors