Orchest is an open source tool for building data pipelines.

Orchest Community

It turns out that this is true for every folder that I create. By default a hidden .ipynb_checkpoints folder gets created in all of them with or without the presence of a jupyter notebook.

image.png

I think this could work. `.ipynb_checkpoints` "pollution" is indeed a bit of a troublemaker. One area I would investigate is whether Meltano has something of an "ignore paths that match this glob pattern" feature

Yes I investigated the same and came across the use of .dbtignore files. in case someone is facing a similar issue, then please have a look at this documentation <https://docs.getdbt.com/reference/dbtignore>

There is actually no user separation in Orchest. Every logged in user has access to the same projects. 

Hope that answers your question!

But what if I want to have people logged in with different accounts have access to a common project. Is that possible?

Oh you mean different accounts from Orchest Cloud? In that case, no.

If you want multiple people to use the same project, then 1 person needs to create an instance on Orchest Cloud and create additional accounts (username + password) through the Settings in Orchest. Then these other people can log in through the URL of the instance (so not through `<http://cloud.orchest.io|cloud.orchest.io>`).

Looks like your instance has a completely full disk, so it's failing to start

yes the status indicator is not up to date because the instance is not responding anymore

So, what should I do? Unless I cannot get into the instance, I cannot delete data

You can either increase the disk size through the cloud panel or you can tell me what you'd like to have deleted and I'll take care of that through the file system, at the moment these single node instances aren't very flexible when it comes to storage, they are free instances mostly meant for  trying out things; the multi node offering has unlimited elastic storage

```psycopg2.OperationalError: FATAL:  database "celery_result_backend" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 47, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 115, in _store_result
    session = self.ResultSession()
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 106, in ResultSession
    return session_manager.session_factory(
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/session.py", line 88, in session_factory
    self.prepare_models(engine)
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/session.py", line 72, in prepare_models
    ResultModelBase.metadata.create_all(engine)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 4864, in create_all
    bind._run_ddl_visitor(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3122, in _run_ddl_visitor
    with self.begin() as conn:
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3038, in begin
    conn = self.connect(close_with_result=close_with_result)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3210, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3289, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3259, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2106, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3256, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  database "celery_result_backend" does not exist

(Background on this error at: <https://sqlalche.me/e/14/e3q8>)```

The problem was solved by creating the database. Not sure why the database `celery_result_backend` was not created on the orchest startup though.

<https://github.com/orchest/orchest/blob/20c8b1555875de4ea93ec4246714d634241ba1df/services/orchest-api/app/app/celery_app.py#L9>

The `celery_result_backend` should be created if it doesn't exist. Maybe the deployment hadn't started the celery app yet which would have attempted creation?

<@U01CBUGK657> can dig deeper. Potentially some race condition where when celery was up the DB was unavailable. Which typically shouldn't happen when `orchest-controller` manages the services.

This is fairly surprising since we automatically deploy multiple Orchest instances on different platforms on releases, e.g. eks, minikube, etc. Haven't had the issue so far.

Could you provide more details about how is the deployment being performed? As Rick mentioned this should indeed not happen

Sorry for the delay. The same problem ocurred to day in another environment.

 The deploy is being done using the following process:
1. Create EKS cluster / Node group using terraform
2. Install calico ( to overcome the EKS limitation of network interfaces )
3. Create iamserviceaccount
4. Install ebs-csi-controller
5. Create namespace orchest
6. Install nginx in the EKS cluster.
7. Deploy orchest-controller.yaml using kubectl apply
8. Deploy orchest-cluster.yaml using kubectl apply.

Anything special (i.e. any changes w.r.t. the original yaml) when it comes to the database? Or any chance that some interaction with calico is going wrong ? Both the `orchest-api` and `celery-worker` will attempt to create the database if it doesn't exist when they start (<https://github.com/orchest/orchest/blob/20c8b1555875de4ea93ec4246714d634241ba1df/services/orchest-api/app/app/__init__.py#L83|link>). The following logs would be interesting (the important part of the logs is likely the head rather than the tail given that the logic which creates the db is run on start)
• `orchest-api`
• `celery-worker`
• `orchest-database`


Hi <@U04NAF9JDAT>, I'm afraid that this kind of mounting will not help in this case, what we are doing for development mode (`--dev`) is that 1) we mount the local directory (i.e. the repo) in the minikube container 2) create a volume which mounts an host path (the repo as seen from the POV of the minikube container)  into the pods which make use of hot reloading. So in this case you are mounting your shared-data directory in the minikube container, but it's never exposed to any Orchest pod.

With this said, Orchest doesn't have an explicit functionality to share data between the host system and the cluster. My go to solution for this, on linux, is to move data in the local directory used by docker for the volume of the minikube container, in particular, in what will be later used as the Orchest `/data` directory, i.e. `/var/lib/docker/volumes/minikube/_data/hostpath-provisioner/orchest/userdir-pvc/data/`

I don't have a mac within reach at the moment but depending on your installation (docker for desktop?) this might not be possible. What I would try is to mount your local directory in the path the minikube container uses for provisioning the Orchest `/data` directory , through the `--mount-string` or `--mount` flag (note that the performance/robustness of this minikube feature can be less than optimal).

All in all, the mount string could look something like this:
`--mount-string="/Users/kevinschaich/shared-data:/var/hostpath-provisioner/orchest/userdir-pvc/data/shared-data"`

To verify the exact path and to verify that the mount is working correctly, you can use `minikube ssh` and check out what's in `/var/hostpath-provisioner/...`

&gt; but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do `ls` from the Jupyter terminal.
Generally speaking Orchest works through 2 directories, the current project directory and the `/data` directory. The `/data` directory is shared across all projects, and will be visible from all steps, jupyter kernels etc., while the `/project` directory mounted to the different steps/kernels etc. is context dependent

Hi <@U04PG7TQHEW> :wave:

Did you follow the installation instructions for GKE from our docs? (<https://docs.orchest.io/en/stable/getting_started/installation.html|link>).

The easiest way to deploy Orchest (if you don't have extensive Kubernetes experience) is to install Orchest on minikube. You can find these instructions in the docs as well (under <https://docs.orchest.io/en/stable/getting_started/installation.html|Linux > minikube&gt;). If you are planning to host Orchest on a cloud VM, then you will need to run a reverse proxy to expose it as well (<https://docs.orchest.io/en/stable/getting_started/installation.html#reverse-proxy|link to docs>).

Hope that helps!

Thank you <@U0184RP9KB8>, I'll check that out.

Hi, could you exec into the orchest webserver and take a look at what's in the `.orchest/user-configurations/jupyterlab/lab` path of the userdir? I'd expect the lock to be there

What's in the `/usr/local/share/jupyter/lab` path of jupyter servers that are and are not failing? I.e. is the lock there for any of them?

If the lock is not there, I'd exec into the failing jupyter servers and take a look at the result of

```userdir_path=/usr/local/share/jupyter/lab
lockdir=$userdir_path/.bootlock
mkdir $lockdir```
I guess the error message will be interesting

jupyter-server-works.png

one thats not working.png

The interesting thing is that the lock is in present in both scenarios:

Logs from a server that's working properly:

Have you tried deleting the lock file? That should pretty much suffice, this snippet of logic

```release_lock() {
    rm -rf $lockdir
}```
is run once the startup logic of the jupyter server is done, perhaps something has gone wrong and a dangling lock has been left around

I haven't tried. Should I do it on the webserver or on each of the containers?

Only once, doesn't matter from which container since those directories are mounted, webserver is ok

I'll try it right now, thanks for the support, as always, Jacopo.

Have you tried restarting the jupyter server pods that are stuck? Now that the lock is there it shouldn't be a problem anymore

are they still stuck waiting on the lock? How many such pods? Could it be that's because there are many jupyter server going through the logic sequentially? I.e. one lock acquire at a time

Currently, 7 of them are stucked in CrashLoopBackoff

I'll kill the deployments and try this process one by one to see if it works

I see, I'd be curious to see what they log before crashing

By deleting the deployments and starting them one by one, I was able to fix this issue. Thank you very much, <@U01CBUGK657>.

Hi <@U04SBC0GWQL> :wave: 

Could you share a little more about your Orchest deployment? What version are you running and did you deploy on minikube, EKS or other? 

Also, if Jupyterlab shows you an error could you copy-paste that in here?

Does this happen with every project or only with a particular project? Anything that could stand out in the environment build script or dependencies?

Hi Bruno, assuming we are talking about multi tenancy, we don't have any good to go material or studies to provide, I can provide a write-up with some suggestions in the following days, not exactly sure how extensive it will be, it's not something we outright toyed with

Ok, I got it. No problem, I believe the suggestions will help us right away. We are running some tests also on this issue, if there is any specific question, I will talk to you. I'll be waiting, thanks

<@U01CBUGK657> basically, today, we have many EKS clusters and we would like to have a single EKS cluster with multiple namespaces with each one containing a full orchest deployment. In this way, we would be able to reduce our current cost. Multitenancy is way ahead of us hehehe

I see, I've sent Bruno a pdf with some (quite limited) notes, not sure if he has forwarded that already, will DM it to you as well

Didn't realize you wanted to go for multiple deployments in different namespaces, I think that kinda lowers the value of the notes I've sent

I think a number of remarks will still be valid, but you will have to take a much deeper look at the `orchest-controller`

Actually no, because in some point in the future we will have to make it multi tenant as well

But for now, I think our problem is more simple

We're thinking about using the following approach. orchest-controller already make possible to customize the `namespace`  and the `orchestHost`
So, we would be able to create multiple orchest deployments in the same EKS.

One problem that we have faced in our internal tests are related to the custom resources that are "global", so we are planning have to remove the definition of those resources from the manifest.

One question that I have is, is possible using this approach to have a single load balancer instead of a multiple load balancers? or even with a single cluster deployment, we would have to create multiple load balancers as well?

&gt; One problem that we have faced in our internal tests are related to the custom resources that are "global", so we are planning have to remove the definition of those resources from the manifest.
do you mean argo &amp; co.?

&gt; One question that I have is, is possible using this approach to have a single load balancer instead of a multiple load balancers? or even with a single cluster deployment, we would have to create multiple load balancers as well?
I find this question difficult to answer, I'm afraid the answer is something along the lines of "yes but it depends on the exact details of what you are interacting with". It's something I've never dabbled with so I can't really provide anything more than a vague guess.

We had some issues with this resource while doing the following simulation:
In a cluster that have a orchest already deployed, we have created a new cluster changing the `namespace` and the `orchestHost` . We were able to deploy the second instance. However, when we tried to delete one of the deployments, it deleted this custom resource and the first orchest cluster stopped working..

The deletion of the namespace failed as well.

<@U01CBUGK657> I think we have misundertood the problem. I think the problem was: we were changing the namespace in the orchest-cluster to a different namespace, however, our orchest-controller.yaml was fixed with the `orchest`  namespace. Therefore, when we tried to delete the orchest-controller.yaml for the second cluster, it deletes resources from the first cluster as well.

By changing the namespaces in the orchest-controller.yaml of the second cluster, we were able to launch two clusters in the same EKS and, later on, delete it without issues.

Hey Rafael, I hadn't had the time to take a look at it yet but I'm glad you found the issue, definitively a time saver on my side :slightly_smiling_face:

<@U048UTACS0J> , one thing that came to my mind, <https://github.com/kubernetes/kubernetes/issues/65551#issuecomment-400909534|custom resource definitions are cluster wide>, so if you'll allow multi tenancy with different Orchest versions you will have to pay attention to not have breaking chances to the CRD on updates

Screen Shot 2023-03-18 at 11.23.43.png

Notebook steps are allowed to be duplicated, because the same Notebook isn't allowed to be used by multiple (&gt;1) steps.

Reason being: the notebook file is changed in-place when it is executed. Thus if two steps run in parallel but point to the same notebook then two steps would try to write to the same file. This doesn't happen for scripts.

Hope that answers your question :wink:

Hi Jericho, is this a cloud or self hosted instance?

When it comes to the jupyterlab setup script being read only, is any juyter build already in progress? What do you see if you scroll back all the way to the bottom of the jupyter build page?

I set auth_enabled to true, and restarted orchest. Unfortunately, now I am unable to access my orchest anymore as I was doing a local port forward.

Edit: A simple change to `<http://localhost:8181/login>` now allows me to access it again!

RE: Jupyter build page

The bottom of the page has a grayed out "Build" key.

That's rather strange, anything interesting that you can see in the network requests being sent? Like to the `/catch/api-proxy/api/sessions` or `most-recent` endpoints

I'm unfamiliar with where to look for those logs

My bad, I meant the browser networking tab, F12 -&gt; network

Looks like it's having trouble reaching the jupyter server

Will try this out tomorrow or the day past tomorrow and come back to you

<@U04VC6E0XL0> I was able to reproduce, seems to be a problem related to the lack of initialization of the setup script, could you run the equivalent of this to initialize it?

```curl '&lt;INSTANCE URL&gt;/async/jupyter-setup-script' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryBAo0zNJGrMdn7muh' \
  --data-raw $'------WebKitFormBoundaryBAo0zNJGrMdn7muh\r\nContent-Disposition: form-data; name="setup_script"\r\n\r\n# placeholder\r\n------WebKitFormBoundaryBAo0zNJGrMdn7muh--\r\n'```
after that you'll be able t edit the setup script and build the custom jupyter image

I've pushed a fix but I can't tell you exactly when we'll make a last release

Hey <@U01CBUGK657>, should I run this while orchest is running?

Yep, it's basically an API call to initialize the setup script

First, an error was reported to us in Jupyter, where it just keeps loading and does not generate a response:

After that, we asked our client to build the Environment again and turn on the session again. However the connect session button is not showing:

Note: the old projects show the session button normally, this behavior is happening only with the new ones.

Hi <@U04R6A3GU73>, I'll take a look at it today

Any chance that this is related to the following?<https://orchest.slack.com/archives/C045TCTCMAP/p1676556260944059>

The first screenshot makes me think that jupyter is failing to start, for whatever reason, and it's probably not related to the environments of the project.

The second screenshot is peculiar, I don't remember us having such a problem (<@U02HPM0TPDK> <@U03K70V292T>?); any chance that the pipeline is not recognized or invalid somehow? There should be one session toggle per pipeline, might very well be that no toggles are there because no pipelines are recognized

Hello <@U01CBUGK657>, <@U02HPM0TPDK>, <@U03K70V292T>. Any news about this subject?

<@U04R6A3GU73> Does the pipeline file (i.e. `.orchest`) exist in the project? If it's placed in the `.ddf` folder, could you try to move it to the root of the project or a folder with a normal name (without the leading dot)?

Hi Reginaldo, what are you getting when running `orchest status`?

Failed to fetch Orchest Cluster status.
The Orchest Cluster named 'cluster-1' in namespace 'orchest' could not be found.

Is the `orchest` namespace there? Did you get any interesting message when running `orchest install` ?

Installing into existing namespace: orchest.
Installing the Orchest Controller to manage the Orchest Cluster...

default     Active  102m
ingress-nginx   Active  102m
kube-node-lease  Active  102m
kube-public    Active  102m
kube-system    Active  102m
orchest      Active  89m

trying to access orchest using chrome results in 404 from nginx

What pods are running in the orchest namespace? Any chance that the minikube mount is failing? The `--dev` flag expects the Orchest repo to be mounted to support hot reloading (see `--mount-string="$(pwd):/orchest-dev-repo" --mount`). Maybe you weren't in the repository directory when starting minikube?

About the mounting, see  <https://docs.orchest.io/en/stable/development/development_workflow.html#cluster-for-development|cluster-for-development> in the same page

I think it is the problem, Jacopo...
I will try again.. thanks for a while...

When installing through the dev flag you indeed need the mount and run the build containers script. The dev flag expects some files in the repo to be present.

The repo is in my home dir, but I don't point correctly to it when run with dev flag....

But... it is possible to install a local image version without use dev flag ?

Could you clarify? Do you mean applying code changes locally without using --dev?

Sorry. Considering you need to use the build-container.sh script when changes are made, and, to avoid issues with hot reloading, couldn't  I just build a set of imagens (for example, scripts/build_container.sh -i orchest-api -t $TAG -o $TAG) and run orchest install without dev flag?

I tested a new minikube mounting using local repo directory and the problem isn't on hot reloading.  Also, the issues remain, there are no resources in orchest namespace and the pods running are: ingress-nginx-admission-create-t98nq, ingress-nginx-admission-patch-sb5v9, ingress-nginx-controller-77669ff58-7868j, coredns-787d4945fb-v57dx, kube-apiserver-minikube, kube-controller-manager-minikube, kube-proxy-cnqhh, kube-scheduler-minikube, storage-provisioner.

&gt; Sorry. Considering you need to use the build-container.sh script when changes are made, and, to avoid issues with hot reloading, couldn't I just build a set of imagens (for example, scripts/build_container.sh -i orchest-api -t $TAG -o $TAG) and run orchest install without dev flag?
Sorry forgot to reply here, yes, you can indeed do that, it' something we did often for development, see <https://docs.orchest.io/en/stable/development/development_workflow.html#redeploying-orchest-after-code-changes>

Thanks for your attention.

In that case, to make a first deploy (without any changes) the steps are:

#Enter inside the root of the orchest repository
cd orchest_local_repo

#Run minikube without --mount-string option
minikube delete
minikube start \
  --cpus max \
  --memory max \
  --addons ingress metrics-server

# Use minikube's Docker daemon:
eval $(minikube -p minikube docker-env)

# Set the *tag* to the latest Orchest version available
export TAG="$(orchest version --latest)"

# Build the minimal set of images for Orchest to run
scripts/build_container.sh -M -t $TAG -o $TAG

# Install orchest
orchest install

So, after any change (like in orchest-api), the steps are:

# Rebuild the images that need it
scripts/build_container.sh -i orchest-api -t $TAG -o $TAG

# Kill the pods of the orchest-api, so that the new image gets used
# when new pod gets automatically deployed.
kubectl delete pods -n orchest -l "<http://app.kubernetes.io/name=orchest-api|app.kubernetes.io/name=orchest-api>"

Note, if Orchest is not installed,  `export TAG="$(orchest version --latest)"` will silently fail

aside from that, this looks correct, but you need to run install before setting the $TAG variable

Just to give a feedback. After to follow the steps discussed (without use dev flag), pods in the namespace orchestra are ok and orchestra seems to be correctly installed. I will conduct some tests. Hot reloading might have been the root of the problems I faced.Thank you, again!

I'm conducting tests to deploy Orchest on a GKE cluster. My cluster is a single node, I applied the necessary yamls to deploy the application, however some PODs were uploaded and others were not, as shown in the images below:

Observing the OrchestComponents, it is seen that the API component was applied, but the deployment and PODS of the API did not go up:

I checked here in orchest-cluster the following status:
```status:
  conditions:
    - event: deploying orchest-database
      lastHeartbeatTime: '2023-04-06T10:37:40Z'
      lastTransitionTime: '2023-04-06T10:37:40Z'
    - event: deploying rabbitmq-server
      lastHeartbeatTime: '2023-04-06T10:38:15Z'
      lastTransitionTime: '2023-04-06T10:38:15Z'
    - event: deploying orchest-api
      lastHeartbeatTime: '2023-04-06T10:38:27Z'
      lastTransitionTime: '2023-04-06T10:38:27Z'
  lastHeartbeatTime: '2023-04-06T10:37:40Z'
  observedHash: 7c97f5f444
  state: Deploying Orchest Control Plane```

It looks like it's still in the deployment process, but it doesn't leave this status.

Could someone help me with this analysis?

Hi <@U04R6A3GU73>, <@U01CBUGK657> will be back on Wednesday

We've never run on GKE, in an ideal world all managed k8s act the same but unfortunately there are usually gotchas.

Most of the idiosyncrasies are related to the storage layer, so maybe check the installation docs for EKS and compare notes to GKE docs to see if everything is properly configured

Hi <@U017WRB10J3>, ok no problems. You don't have any internal document with GKE context right?

Nope, we hadn't ever had a need to deploy to GKE yet

Uh cool, I'll take a look at the storage part. Thanks a lot for the feedback.

<@U02NC2KJWKD> container images likely are consuming the most files. Do you have access to the local container image registry (`docker image ls`)?

`docker system prune` should help. It removes all dangling Docker images and other stuff that can be taking some space there

Hi Allan,

I'm assuming that we are talking about multiple auth users sharing the same Orchest instance, and not multiple instances deployed in the same cluster

My advice would be to either:
•  keep going with an approach similar to how ssh keys were/are implemented based on the `orchest-api`  `AuthUser` model to essentially tie some resources to the auth user itself, but strengthen the security around how secrets are accessed so that it's tied to the user actually being logged in rather than just based on the user id
• use k8s RBAC to control access to k8s secrets. Define a role for each tenant that grants them read access to their own secrets, then bind the role to all pods that make use of said secrets. This isn't something I've personally ever toyed with so take this with a big pinch of salt
• use a third party solution (like hashicorp vault maybe?) to pretty much do point 1 but better

Orchest was thought to be deployed as a sort of cell-based architecture, each customer would have it's own Orchest instance, with all users belonging to the same instance being considered at the same security/access level, so transforming Orchest into a multi tenant solution will be an uphill battle, you can indeed use the `AuthUser` model to essentially implement something akin to row level security but there will be adjustments to be made, and there is the whole angle related to the fact that you are running user code in steps, builds, etc. which could try to circumvent your security and call Orchest services like the api, the db etc. directly, so networking (calico?) and RBAC work will be needed

Hi Triet, I think you should be looking into the files at `services/orchest-webserver/client/src/pipeline-view/file-manager/`

That particular docker image is something we used in the pre k8s version of Orchest

I see. Thank you. In the current version, you don’t use <https://github.com/filebrowser/filebrowser|filebrowser> anymore, right?

Hi <@U01CBUGK657> thanks for quick response. so I should run on minikube-ip 's 8000 port?

Screenshot 2023-04-28 165330.png

As in <https://docs.orchest.io/en/stable/getting_started/installation.html#> run minikube ip, you'll get an ip to which you'll be able to reach the applicatio at ip:port, port should be 8000

You can also change your hosts file to map that ip to something easier to remember, e.g.

`192.168.49.2    <http://localorchest.io|localorchest.io>`

how to change? requesting you to please share some document

You can add such an entry to `/etc/hosts`, the ip must be the one returned by `minikube ip`

added, but it shows website not available

when visiting <http://localorchest.io|localorchest.io>

what are you getting from running `minikube ip`?

And what are you getting when going at `192.168.49.2:8000`? *http*

Screenshot 2023-04-28 171218.png

Screenshot 2023-04-28 171932.png

Even with http also it is giving the same result

could you try on port 80? :thinking_face:
`<http://192.168.49.2>`
or
`<http://192.168.49.2:8000/projects>`

Is the ingress addon enabled?  `minikube addons enable ingress`

Did you use the convenience install script in the installation docs?

Are you running through windows  + Windows Subsystem for Linux?

```{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}```


This is a response from kubernertes, what endpoint are you querying?

btw, here's some internal notes about how we proceeded when installing Orchest in windows through wsl, it's not officially supported but imo the issues you are having are solvable, although I'm not sure how much I can help without much more context around your setup

• open a terminal in windows, run `wsl --install`
• reboot
• verify wsl is working, `wsl --install -d Ubuntu`
• install docker for desktop in windows, i.e. not through the wsl terminal. Once installed you can set it to use the wsl + the distribution of your choice. Follow these instructions: <https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers>
• once that’s done get into the terminal and run wsl with `wsl`
• install k9s, pip, orchest-cli and then install orchest
```# k9s
wget <https://github.com/derailed/k9s/releases/download/v0.25.18/k9s_Linux_x86_64.tar.gz>
tar -xvzf k9s_Linux_x86_64.tar.gz
sudo mv k9s /usr/bin


sudo apt-get update &amp;&amp; sudo apt-get install orchest-cli -y
pip3 install orchest-cli
orchest install
# Ingress
kubectl apply -f <https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.1/deploy/static/provider/cloud/deploy.yaml>```

I made the json adjustment to enable authentication. When I go to restart to apply the change I get this error:

Do you have any idea what this error could be?

For better context: I uploaded a GKE cluster and deployed the application in two different namespaces, each namespace is attached to a node group. In a namespace application I didn't have this problem, it's happening in the other one.

Anything interesting in the browser debugging tools? Both console messages and network request response code and payload

Any chance that the namespace for which things are working is the default namespace?

Can you check if the values of the env variables `ORCHEST_NAMESPACE` and `ORCHEST_CLUSTER` are set correctly for the `orchest-api` pod part of the namespace for which things are failing?

Should also check the following pods logs when making a restart request which fails:
• `orchest-api`
• `orchest-webserver`
• `orchest-controller`


Hello Jacopo. Thank you for the informations. Looking at the logs I found some inconsistencies and managed to do a reset and the mentioned problem was solved.

However, I have another problem and I would like some advice from you:

In this same described scenario, a cluster running two Orchests with two different namespaces, and in this cluster I have two NodePools. Through Labels I attached the main deployments to each corresponding NodePool.

However, when I go to build some environment I noticed that it uploads a POD called "image-build-task" and this POD is not being attached to the correct NodePool, causing an error.

Is there any way for this problem to be resolved? If you need more details, I'll let you know in this thread.

&gt; Hello Jacopo. Thank you for the informations. Looking at the logs I found some inconsistencies and managed to do a reset and the mentioned problem was solved.
happy to hear that!

&gt; However, when I go to build some environment I noticed that it uploads a POD called "image-build-task" and this POD is not being attached to the correct NodePool, causing an error.
Take a look at `_get_image_build_manifest` in `services/orchest-api/app/app/core/image_utils.py` , you'll see the `pod_scheduling.modify_image_builder_pod_scheduling_behaviour(manifest)` call at the end of the function, the `services/orchest-api/app/app/core/pod_scheduling.py` module is in charge of doing changes to manifests so that scheduling happens in a way we like, in this particular case, it'll look for the `WORKER_PLANE_SELECTOR` env variable of the orchest-api and celery-worker

to adjust the behaviour for multi tenancy you'll likely need to do some minor changes around these variables

Hi, I might have time to look into that tomorrow

At a first glance, I'd make sure something isn't going wrong with the installation when it comes to the Orchest version, you are installing with

`export VERSION="v2023.01.8"`

but you report `v2023.04.2` from the settings page. Might be worth taking a closer look into the yamls

Does the cluster have an ingress? Like nginx? <https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx>

The coredns error doesn't seem something on our end, but tomorrow I'll try to look further into this

Hi 小饭, if you could provide a a short script or summary of how you deployed kind + Orchest it'll likely help me in finding a solution

Hello guys, we're implementing some security measures in our orchest cluster.

I was not the one that have created our deployment process in the past...

We were thinking about using AWS WAF to secure it. However, since our load balancer is of type network (layer 4), WAF is not compatible. Is there a reason for orchest to create a Network Load Balancer instead of a Application Load Balancer?

Here is our manifest:
```apiVersion: v1
kind: Service
metadata:
  annotations:
    <http://service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout|service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout>: "60"
    <http://service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled|service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled>: "true"
    <http://service.beta.kubernetes.io/aws-load-balancer-ssl-cert|service.beta.kubernetes.io/aws-load-balancer-ssl-cert>: ....
    <http://service.beta.kubernetes.io/aws-load-balancer-ssl-ports|service.beta.kubernetes.io/aws-load-balancer-ssl-ports>: https
    <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: nlb
    <http://service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags|service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags>: environment=....
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: controller
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ingress-nginx
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: ingress-nginx
    <http://app.kubernetes.io/part-of|app.kubernetes.io/part-of>: ingress-nginx
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.3.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: tohttps
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: http
  selector:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: controller
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ingress-nginx
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: ingress-nginx
  type: LoadBalancer```


Hi Rafael, in principle I don't see why you couldn't go with an ALB rather than a NLB assuming the due diligence checks take place. You might have to double check the controller for any quirky behaviour around balancers (not that I know of, but I wasn't fully involved with the controller) and the logs streaming since they go through websocket

I didn't have the chance to work with WAF so I can't really help with any knowledge in that direction sadly

I was able to workaround this, it's an issue of the ingress-controller itself, it seems that it's not compatible (directly at least) with application load balancers, just classic and network load balancers. However, we were able to use a load balancer by changing the service type to NodePort and forwarding requests from an existing load balancer to the node port.

Happy  to hear you found a workaround :slightly_smiling_face:

Screenshot from 2023-07-17 13-09-44.png

I figured it somewhat out myself. apparently it came down to poorly configured docker driver in minikube. i change driver to --driver=kvm2 and everything works great. :slightly_smiling_face:

Hello. how are you?
Nice to meet you.
I am interested in this remote position.
I am a full stack developer who has 7 years of experience in web and mobile development.
I would like to take part in your senior team and contribute with my experienced skils.
You can check my profile here.
<https://www.linkedin.com/in/david-rui>
I am looking forward to hearing from you.
Thank you.
David.

Hello everyone. I am trying to build an Orchest webserver from source code. However, in the Dockerfile, it can not find the /pnpm folder. Can you help me?

error.txt

When building the Orchest Webserver from the source code. It threw the errors as the attached file. Could you please help me resolve this problem?

Hello Everyone, Good day to you.

I wanted to inform that; We currently have a few *AWS Solution Architect Professional Certified* and *Terraform Certified* DevOps Engineers available on our team.

If you have any ongoing or upcoming requirements where our expertise could be beneficial, please feel free to let me know.

Immediate Available | Remote | Best C2C contract rate | For Hire

Hi everyone, Where can i find the build & install documentation for orchest community in windows? The link in the github README.md is not working.