Exosphere

Problem/Opportunity Statement On the Jetstream2 project, we have been receiving an increasing number of tickets recently from users who claim to have performed an action on their instance, such as a shelve/unshelve or delete action, that hangs ... [More] indefinitely. When we go to investigate, we find that these action requests never actually hit our API endpoint. From the user perspective, when they hit the Shelve button, the instance changes to "Shelving". Then the user leaves either leaves the page entirely thinking the action went through or they leave the page open but don't refresh the page and the state stays on shelving for a long time, appearing stuck. Once a user encounters this issue once, it appears to occur repeatedly for all of their actions until they reauthenticate. By this I mean clicking the "Remove All Allocations" button, then reauthenticating and re-adding their project to their Exosphere client. I don't know the underlying mechanics of Exosphere well enough to understand why some user requests are not hitting our API endpoint successfully, nor am I sure how to replicate this. I am also unsure why re-authenticating fixes the issue. Whether this is some sort of de-sync or stale credentials issue is besides the point for this issue. Maybe that needs its own separate issue. What would success / a fix look like? For the sake of scope for this issue, I think a fix would look like Exosphere not changing the state of an instance until it has confirmed that the state has actually changed. For example, if a shelve request fails to go through, the state of the instance won't actually change from active -> shelving -> shelved on the jetstream2 side; it will stay in active. If this were the case, when a user tries to shelve an instance but it doesn't succeed, they will be able to see that their instance is still in an active state and did not change. Rather than thinking they successfully told an instance to shelve and being suprised when they return later to find the instance was active and burning SUs that entire time, they can recognize something is not working as expected and reach out to get support. [Less]

Depends on !1130 (merged). The first two commits here are that in that MR. Overview Users on Jetstream2 and other OpenStack clouds can have object storage in their allocation, but Exosphere can't touch it. Creating a container, uploading a file ... [More] , sharing it, or cleaning up all mean dropping to the CLI or rclone. This adds a browser UI for the everyday cases, on top of the Swift client from !1130 (merged). When the catalog has an object-store endpoint, the project page gets an Object Storage tile (same pattern as the Shares tile with manila). From there you can: List containers with object count and size, create new ones, and delete them. Deleting a non-empty container empties it first, recursively but bounded, behind a confirmation. Browse a container as folders using delimiter=/ and prefix, with breadcrumbs. Listings paginate past Swift's 10,000-item cap. Upload and download files. Uploads go through a queue with per-file status. There is no progress bar because elm/http can't report progress without a JS port, so I preferred honest statuses over a fake bar. Files over about 100 MiB are rejected up front with a pointer to the CLI, since the whole file passes through browser memory. Copy and move objects. Copy is server-side via X-Copy-From. Move only deletes the source after the copy succeeds. Create folders (a zero-byte application/directory placeholder, the usual Swift convention). Manage container access. A public/private toggle covers the common case; a structured grants editor and a raw X-Container-Read/X-Container-Write field sit under an "advanced" disclosure. Grant writes merge with what's already there instead of overwriting, and the UI re-reads the container HEAD after a change rather than assuming it worked. Making a container private sends X-Remove-Container-Read so the old grant is actually revoked. Connect an S3 client. A tile shows the s3 endpoint if the catalog has one, the user's EC2 credentials from Keystone (secret masked until clicked), and a ready rclone config. If the cloud has no s3 endpoint the tile says so. There are user docs in docs/object-storage.md, with an operator section and a table of what still needs the CLI. One fix worth calling out: Swift responses include Last-Modified but no Cache-Control, so browsers cache listing bodies. The listing you fetch right after an upload can come from cache and not show the new object until a hard reload. All Swift reads therefore carry a throwaway t query parameter, the same trick Rest.AppVersion uses for version.json. Not included, listed in the docs: large objects (SLO/DLO), object versioning, object metadata editing, TempURL links, and talking the S3 protocol directly. The Rest module is kept backend-agnostic so an S3 backend could be added behind the same UI later. For deployers: all Swift traffic goes through the CORS proxy, and the proxy config needs two changes to support this feature fully. The expose-headers list must include the Swift headers this UI reads (X-Container-Read, X-Container-Write, X-Container-Bytes-Used, X-Container-Object-Count, X-Timestamp, ETag, Content-Disposition), and the body size limit must be raised above nginx's default 1 MiB or uploads fail. Today the try-dev proxy exposes only X-Subject-Token, which is why the container Info card on Jetstream2 currently falls back to showing no usage data. How to Test You need a cloud with object storage and a proxy that accepts your origin. For a local build, try-dev.exosphere.app/proxy works (production proxies reject localhost origins). Log into a project on a cloud with an object-store catalog entry. On Jetstream2, federated accounts need an admin-set OpenStack password to log into a local build, since there is no application-credential login yet. Open the project page and click the Object Storage tile. Create a container, browse into it, make a folder, upload some files. Try a file over 100 MiB to see the size guard. Download something, copy and move something, bulk delete. Open manage access: make the container public, check the public link, add a grant under advanced, then make it private again. Look at the S3 tile on a cloud with and without an s3 endpoint. npm test covers the decoders, ACL parsing and serialization, pagination stitching, path building, and the LocalStorage migration (366 tests, up from 203 on !1130 (merged)). I have run the full flow against devstack Swift (including 1, 25, and 100 MiB uploads, ACLs, bulk delete, and CORS preflight through an nginx proxy) and the main flow against Jetstream2's Ceph RGW: listing, browsing, upload, download, and the detail page all work there. What I could not yet verify on RGW: which Swift ACL grant forms it honors, the public URL shape, the bulk-delete response format, and EC2 credential handling against its Keystone. Help from anyone with RGW access is welcome. Screenshots [Less]

💯 Thanks, @JulianGonzalezR, I believe the latest commits close all my open points! 🚀 Happy with this from my side & well done on the swell feature!

I don't think people using curl is the problem here. I suspect we need CORS here to protect against malicious code embedded in other websites, right? I haven't worked through all the implications yet, so I might be wrong and/or too paranoid. 😅

I think we can get rid of it for this specific endpoint. It sets no cookies or logged-in session and the passphrase goes in the POST body, so the allowlist isn't protecting much. If we ever add session auth we'd have to add it back again.

🙌 Thank you so much for those changes, @JulianGonzalezR! 👌 They're looking really good, I liked the folding away of the S3 connect at the top of the page & seeing the new copy icon everywhere. I've got a few small pieces of feedback & used ... [More] my agent to consolidate them & detect some other stragglers. Please shout if you need more detail on any or your can't reproduce them? Item Status Project overview tile spins forever ✅ Fixed. Fixed in c5457819. Validated against Jetstream2 IU. Info toggle tip should be a public/private status badge ☑️ Badge added in 87beb6ed, which treats X-Container-Bytes-Used or X-Container-Object-Count as evidence the headers survived the proxy and renders no badge rather than guessing.✏️ accessWord was not moved onto containerAccess, so the Info strip below the badge still reports access Private from the same stripped response. Validated: badge silent, Info card asserting Private two lines down. Promote the create form to a create page and list it in the Create dropdown ☑️ New Page.ObjectStorageContainerCreate in 6e68f18e, with a /createcontainer route consistent with createshare and createvolume, a Create dropdown entry, and the house Validation.invalidMessage widget.✏️ The new dropdown entry at View.elm:567 gates on project.endpoints.swift alone, while the overview tile uses objectStorageTileVisible, which also requires experimentalFeaturesEnabled — so with the flag off the tile hides and the Create entry does not. object deserves a localization ☑️ objectStoreObject added in e30ae7ad and wired through the flags decoder, defaults, types.d.ts, both environment configs and the NoHardcodedLocalizedStrings rule. Default remains object, and no hardcoded occurrences remain.✏️ It is required in localizationDecoder but missing from the localization block in docs/config-options.md. Folder placeholder shows as an empty row at the top of its listing ✅ Fixed. Fixed in 0c937aaa, narrowed in 35493919 so only a zero-byte object counts as a placeholder and a real object named docs/ stays visible. The page-fullness test still runs on the unfiltered page, so pagination does not stop a row early. Folder/subfolder text is cut off at the bottom 🔍 Open. Cause identified: the breadcrumb sits in an Element.el carrying maximum 600, scrollbarX and clipX at ObjectStorageContainerDetail.elm:1589, and elm-ui's horizontal scrollbar clips descenders on an auto-height row. A short path hides it, which is why it does not reproduce on every container. Recommendation: Settle by dropping the scroll container or giving the row an explicit height. Container delete confirmation 🔍 Open. The assembled sentence reads This permanently deletes all 42 objects inside it, then the container itself, not just the container. Recommendation: Settle by dropping the trailing clause. Any 403 on the OS-EC2 list is presented as proof of an application credential 🔍 Open. s3CredentialPanelDecision infers S3CredentialPanelApplicationCredentialForbidden from the status alone, so an operator policy denial gets a confidently wrong explanation. project.secret already distinguishes ApplicationCredential from NoProjectSecret. Recommendation: Settle by requiring both signals. User docs omit the experimental-features gate 🔍 Open. docs/object-storage.md says the tile appears when the service is available, but objectStorageTileVisible also requires experimentalFeaturesEnabled, and neither that file nor docs/config-options.md mentions it. Container detail title bar does not fill the width ☑️ Fixed in 87beb6ed; the header is now icon, localized noun, container name, status badge and Actions.☑️ Also removed a separate defect found alongside it: the container name was passed through toTitleCase, so backups displayed as Backups. Loading containers… should say objects ✅ Fixed. Fixed in 0c937aaa, using the objectStoreObject localization. S3 client tile sits far below the container list ✅ Fixed. Moved above the list and made collapsible in ae9792bc. Validated live, including the no-S3-endpoint message on Jetstream2 IU. clipboardCopyButton duplicated across two pages ✅ Fixed. Fixed in 6a4ed68d as CopyableText.copyButton, with copyTextAttributes exposed for the dropdown-item case. Both page-local copies are gone. maskedScriptBlock reimplements copyableScript ✅ Fixed. Fixed in 6a4ed68d as proposed: copyableScriptMasked is the implementation and copyableScript passes the same string as display and clipboard. Adopt the new copy icon rather than run two in parallel ✅ Fixed. Adopted in 6a4ed68d; the shared accessory draws the Feather clipboard icon, so existing copyableText callers pick it up. Eight comments truncated mid-sentence ✅ Fixed. All eight completed in 0a1336c0, leftover em dashes replaced in 9d5d9263. A re-scan of the object storage modules found no remaining fragments. Three docstrings contradict their own code ✏️ ObjectStorage.elm:732 says bulk-delete success requires both a 2xx status and no per-object Errors, but checks only the status. Both call sites do the List.isEmpty errors test themselves, so the behaviour is right and the docstring is wrong.✏️ DataList.elm:322 says non-selectable rows render BLANK space (of the same width, so columns stay aligned), but rowView drops the checkbox column entirely, as its own inline comment states.✏️ Route.elm:752 says the container-detail parser MUST precede the container-list parser, but the two consume different segment counts and can never both match. Several docstrings describe the change rather than the code ✏️ ObjectStorageContainerDetail.elm:1163: making a container private no longer flips immediately.✏️ ObjectStorageContainerDetail.elm:1841: Replaces the old full-URL "Public link:" row + the duplicate open-in-new-tab anchor.✏️ ObjectStorageContainerDetail.elm:2018: replacing the earlier PROVISIONAL String.contains ".r:*" substring check. Now populated by the HEAD-container read.✏️ IconButton.elm:55: clickableIcon keeps its historical 22px default for existing callers.ℹ️ All four are introduced by this branch and stop making sense once it merges. Container name placeholder hardcodes the default localized noun 🔍 Open. ObjectStorageContainerCreate.elm:109 still uses the literal my-container, carried over unchanged when the form moved to its own page. (The NoHardcodedLocalizedStrings rule matches only a localization's default value and cannot see it.) [Less]

Problem/Opportunity Statement What would success / a fix look like?

Overview Fixes #262. Continues !1101 (closed) by Julian Pistorius (thanks Julian); his original commit opens this branch. The direct application credential login is kept, and the manual ID and secret form is replaced by the credential file. You can ... [More] now log in by dropping (or browsing for, or pasting) an openrc.sh or a clouds.yaml on the OpenStack login screen. The file is recognized by its content, not its name. A clouds.yaml with one cloud logs you in; one with several clouds lets you pick which ones. Files that are not one of the two formats are refused with a message. If the file names a region, that region is used and the region question is skipped. Logins that still need a region are queued and answered one by one, and the region question can be cancelled without stranding the login queue. Logging in to a project you already have says so on the login screen instead of doing nothing. The OpenRC parser follows shell semantics: the last assignment of a variable wins, trailing comments stay out of values, and a file that names an application credential but leaves out a piece (the secret, the auth URL, or a blank value) is refused with a message rather than guessed at. Re-login matches a project by cloud as well as by project UUID, so a credential for one cloud can no longer stamp its token and secret onto a same-numbered project on another cloud. Main files: src/OpenStack/CloudsYaml.elm, src/OpenStack/CredentialFile.elm, src/OpenStack/OpenRc.elm (now a pure parser), src/Page/LoginOpenstack.elm, src/Page/SelectProjectRegions.elm, and the region and identity handling in src/State/State.elm, src/Rest/Keystone.elm, and src/Helpers/Url.elm. How to Test Create an application credential in Horizon (Identity, Application Credentials) and download the clouds.yaml it offers. npm start, then Add Allocation, OpenStack. Drop the file on the screen. It should show the project and Keystone URL it read. Log In should land you in that project. Rename the file and try again: same result. Drop the same file once more after logging in: an info notice says the project is already added. Try a clouds.yaml with several clouds: you get a picker and can log in to more than one. Paste an OpenRC file with an application credential in the text box: it should log you in too. On a multi-region cloud you get the region question, with a Cancel that abandons that login cleanly. Drop a random text file: it should be refused with a message. npm run test covers the parsers, the screen logic, and the region queue. Screenshots [Less]

Overview Fixes part of #862 (Guacamole only; JupyterLab and other interactions can follow the same pattern) and part of #1039 (the browser-interaction slice, on clouds with routable IPv6; native SSH is out of scope here). Adds a second way for the ... [More] browser to reach Guacamole on an instance: directly over HTTPS at the instance's own IP address, with a Let's Encrypt certificate issued for that IP. The existing User Application Proxy route is unchanged and stays the default; clouds opt in with a new directGuacamole boolean in cloud_configs.js. All Guacamole URL building now goes through one resolver, Helpers.GuacamoleEndpoint.resolve: Direct mode on and the instance has a floating IP: connect to that. Direct mode on, no floating IP, but a globally routable IPv6 fixed address (2000::/3 only, so never link-local or unique-local): connect to that, bracketed. Otherwise: the proxy, exactly as today. Instances launched before a cloud opts in keep using the proxy. The exoGuac metadata is bumped to v2 with a tls field; a missing field decodes as false, so nothing strands. On the instance, a new caddy Ansible role fronts Guacamole with TLS. It discovers the floating IP from the OpenStack metadata service (retrying, since Exosphere attaches it after the server goes active) and issues a certificate for every global IPv6 address it finds. The Caddyfile template came off a working Jetstream2 prototype (both address families, real Let's Encrypt IP certs, Exosphere's Guacamole stack behind it) and its comments record the traps, notably fallback_sni: browsers send no SNI for IP-literal URLs, so a NAT'd floating IP fails TLS without it. Caddy also answers CORS for the token POST, allowing only the Exosphere origin that launched the instance (passed as an extra var), because the proxy silently provided CORS before. Two things reviewers should weigh in on: defaultRules gains TCP 443 ingress for IPv4 and IPv6 on every cloud. The v4 rule is redundant with the existing expose-all rule; the v6 rule is genuinely new exposure and is the real question. Let's Encrypt only issues IP certificates under the shortlived profile (about six days), so the instance depends on continuous renewal, and the ACME challenge is TLS-ALPN-01 on 443, verified cold in an unattended end-to-end launch on Jetstream2: instance create to valid HTTPS Guacamole in 2m12s, certificates for both the floating IPv4 and the global IPv6, zero Ansible failures. One follow-up decision: Guacamole's plaintext port 49528 stays world-reachable in direct mode (pre-existing behavior from proxy mode); a direct-TLS instance does not need it published. How to Test Run the usual suite (format, analyse, review, tests, build). Then: Regression, the important half: on a proxy cloud without directGuacamole, Web Shell and desktop URLs, and both unavailable messages, must be byte-identical to master, including for instances launched before this branch. Direct mode: set "directGuacamole": true on a cloud, launch an instance with Guacamole. After Caddy gets its certificate (a minute or two; the interaction retries until then), Web Shell opens at https:// [Less]

For future enhancement: We may need to be more flexible, and allow other origins. Ideally somebody should be able to change this after launching the instance. Fortunately Caddy has an admin API! https://caddyserver.com/docs/api