Expected behavior:
Following the documentation at https://goteleport.com/docs/management/operations/backup-restore/#migrating-backends should work for migrating users to a new storage backend such as postgres even if trusted clusters are configured.
Current behavior:
If trusted clusters are configured, the bootstrap process fails with the following error:
ERROR: initialization failed
refusing to bootstrap backend
expected PEM-encoded block
Removing the kind: cert_authority entries for all trusted clusters manually from the state file resolves this issue. As trusted clusters are not migrated, I would expect these to not need to be manually removed during a migration. If this is a requirement, I would expect it to be called out in the documentation.
Bug details:
- Teleport version: v15 (observed on v14 too)
- Recreation steps
Attempt to follow the migration documentation on a cluster with trusted clusters configured
- Debug logs
[root@Test-Cluster-Node1-US-East ~]# teleport start --bootstrap state.yaml -d
2024-02-14T16:01:46Z WARN ca_signing_algo config option is deprecated and will be removed in a future release, Teleport defaults to rsa-sha2-512. config/configuration.go:497
2024-02-14T16:01:46Z INFO Using license from /etc/licenses/super-license.pem expires at 2122-02-18 23:59:59 +0000 UTC,supports kubernetes,supports application access,supports database access,supports desktop access. process/license.go:30
2024-02-14T16:01:46Z DEBU [SQLITE] Connected to: file:/root/fresh/proc/sqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate, poll stream period: 1s lite/lite.go:258
2024-02-14T16:01:46Z DEBU [SQLITE] journal_mode=delete, synchronous=2, busy_timeout=10000 lite/lite.go:309
2024-02-14T16:01:47Z DEBU [PROC:1] Adding service to supervisor. pid:15736.1 service:update.aws-oidc.deploy.service service/supervisor.go:223
2024-02-14T16:01:47Z DEBU [PROC:1] Using dynamodb backend. pid:15736.1 service/service.go:5456
2024-02-14T16:01:47Z INFO [DYNAMODB] Initializing backend. Table: "test-cluster1", poll streams every 1s. dynamo/dynamodbbk.go:236
2024-02-14T16:01:47Z WARN [CLOUDLABE] Could not fetch EC2 instance's tags, please ensure 'allow instance tags in metadata' is enabled on the instance. labels/cloud.go:141
2024-02-14T16:01:48Z DEBU [DYNAMODB] AWS session is created. dynamo/dynamodbbk.go:365
2024-02-14T16:01:48Z DEBU [DYNAMODB] Found latest event stream arn:aws:dynamodb:us-east-1:174321026473:table/test-cluster1/stream/2021-01-15T15:01:35.226. dynamo/shards.go:83
2024-02-14T16:01:48Z DEBU [BUFFER] Add Watcher(name=external_audit_storage, prefixes=/external_audit_storage/cluster, capacity=1024, size=0). backend/buffer.go:265
2024-02-14T16:01:49Z INFO [S3] Setting up bucket "schisaport-test-cluster1", sessions path "/records" in region "us-east-1". s3sessions/s3handler.go:221
2024-02-14T16:01:49Z INFO [S3] Setup bucket "schisaport-test-cluster1" completed. duration:686.681514ms s3sessions/s3handler.go:225
2024-02-14T16:01:50Z INFO [DYNAMODB] Initializing event backend. dynamoevents/dynamoevents.go:256
2024-02-14T16:01:51Z DEBU Creating in-memory backend for [auth]. accesspoint/accesspoint.go:92
2024-02-14T16:01:52Z INFO [AUTH] Applying 84 bootstrap resources (first initialization) auth/init.go:333
ERROR REPORT:
Original Error: *trace.BadParameterError expected PEM-encoded block
Stack Trace:
github.com/gravitational/teleport/lib/utils/certs.go:111 github.com/gravitational/teleport/lib/utils.ParsePrivateKeyPEM
github.com/gravitational/teleport/lib/auth/keystore/software.go:85 github.com/gravitational/teleport/lib/auth/keystore.(*softwareKeyStore).getSignerWithoutPublicKey
github.com/gravitational/teleport/lib/auth/keystore/software.go:81 github.com/gravitational/teleport/lib/auth/keystore.(*softwareKeyStore).getSigner
github.com/gravitational/teleport/lib/auth/keystore/manager.go:202 github.com/gravitational/teleport/lib/auth/keystore.(*Manager).getSSHSigner
github.com/gravitational/teleport/lib/auth/keystore/manager.go:177 github.com/gravitational/teleport/lib/auth/keystore.(*Manager).GetSSHSigner
github.com/gravitational/teleport/lib/auth/init.go:944 github.com/gravitational/teleport/lib/auth.checkResourceConsistency
github.com/gravitational/teleport/lib/auth/init.go:334 github.com/gravitational/teleport/lib/auth.initCluster
github.com/gravitational/teleport/lib/auth/init.go:310 github.com/gravitational/teleport/lib/auth.Init.func1
github.com/gravitational/teleport/lib/backend/helpers.go:222 github.com/gravitational/teleport/lib/backend.RunWhileLocked
github.com/gravitational/teleport/lib/auth/init.go:301 github.com/gravitational/teleport/lib/auth.Init
github.com/gravitational/teleport/lib/service/service.go:1805 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService
github.com/gravitational/teleport/lib/service/service.go:1112 github.com/gravitational/teleport/lib/service.NewTeleport
github.com/gravitational/teleport/e/tool/teleport/process/process.go:59 github.com/gravitational/teleport/e/tool/teleport/process.NewTeleport
github.com/gravitational/teleport/lib/service/service.go:696 github.com/gravitational/teleport/lib/service.Run
github.com/gravitational/teleport/e/tool/teleport/main.go:35 main.main
runtime/proc.go:267 runtime.main
runtime/asm_amd64.s:1650 runtime.goexit
User Message: initialization failed
refusing to bootstrap backend
expected PEM-encoded block
Expected behavior:
Following the documentation at https://goteleport.com/docs/management/operations/backup-restore/#migrating-backends should work for migrating users to a new storage backend such as postgres even if trusted clusters are configured.
Current behavior:
If trusted clusters are configured, the bootstrap process fails with the following error:
Removing the
kind: cert_authorityentries for all trusted clusters manually from the state file resolves this issue. As trusted clusters are not migrated, I would expect these to not need to be manually removed during a migration. If this is a requirement, I would expect it to be called out in the documentation.Bug details:
Attempt to follow the migration documentation on a cluster with trusted clusters configured