Azure Container Apps: Internal Server Error on provisioning ACA Environment in existing VNet

Product:

Azure Container Apps, Azure Virtual Networks

Problem:

As I was trying to set up ACA in an existing lab VNet, I discovered that some of my operations for creating the ACA Environment were failing with "Internal Server" error. 

Fortunately I've been through this type of obtuse error messaging from Azure before, so I was able to quickly walk back to what seemed to be causing my issue. 

Powershell received the following error:

New-AzContainerAppManagedEnv @EnvArgs
New-AzContainerAppManagedEnv : Internal server error occurred.  correlation ID: <snip>
At line:1 char:1
+ New-AzContainerAppManagedEnv @EnvArgs
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: ({ SubscriptionI...    }
  }
} }:<>f__AnonymousType16`4) [New-AzContainer..._CreateExpanded], Exception
    + FullyQualifiedErrorId : InternalServerError,Microsoft.Azure.PowerShell.Cmdlets.App.Cmdlets.NewAzContainerAppManagedEnv_CreateExp
   anded

ARM deployments received multiple of the following error and then timed out: 

    "properties": {

        "statusCode": "InternalServerError",

        "serviceRequestId": null,

        "statusMessage": "{\"error\":{\"code\":\"InternalServerError\",\"message\":\"Internal server error occurred. correlation ID: <snip>\"}}",

        "eventCategory": "Administrative",

        "entity": "/subscriptions/<snip>/resourcegroups/<rgname>/providers/Microsoft.App/managedEnvironments/<envname>",

        "message": "Microsoft.App/managedEnvironments/write",

    },

 

 

Solution: 

Remove IPv6 address spaces from the VNet. 

 

HTH.  

Terraform quick view of resource schema

 I was getting tired of having to go to Hashicorp's website to view provider arguments lists, because SURELY the arguments list is in the code somewhere handy. As it turns out, it is.

I made a quick bash function to export the schema for a given resource provider

tfschema () {
    if [ $# -ne 2 ]
    then
        echo -e "This function requires exactly two arguments, in order: providername resourcename "
        echo "Try: tfschema  registry.terraform.io/hashicorp/vsphere vsphere_virtual_machine"
        echo "It is required to have already run terraform init with HCL for the provider used in the first argument."
        echo "Try again. You have 1 life remaining. Exiting..."
        return
    fi
    if ! which terraform >/dev/null 2>&1
    then
        echo "terraform not found. Exiting..."
        return
    fi
    
    terraform providers schema --json|jq '.provider_schemas["'"$1"'"].resource_schemas["'"$2"'"].block| .attributes, .block_types|values'
}

Quick Screenshot:

Terraform Azure Devops hosted pipelines

 In setting up my homelab using AZDO, I wanted to run some builds and steps using the Azure Hosted Agent. 

The documentation is a little unclear on this so I imported resources as follows. 

If codeblock output is clipped by css, don't worry, the code is there. Not working on that tonight as I have homelab to set up. 

For given TF code:

resource "azuredevops_agent_pool" "pipelinesagent" {

  name           = "Azure Pipelines"

  auto_provision = true

}

resource "azuredevops_agent_queue" "pipelinesqueue" {

  project_id    = azuredevops_project.project.id

  agent_pool_id = azuredevops_agent_pool.pipelinesagent.id

}

I found the azuredevops_agent_pool ID and the azuredevops_agent_queue ID as follows:

az pipelines queue list

// There were a lot of queues, but I am interested in the last in the output: 

// guids replaced with <guidhere>, get your own.

// 

[

{

    "id":  45,

    "name":  "Azure Pipelines",

    "pool":  {

                 "id":  9,

                 "isHosted":  true,

                 "isLegacy":  false,

                 "name":  "Azure Pipelines",

                 "options":  "none",

                 "poolType":  "automation",

                 "scope":  "<guidhere>",

                 "size":  1

             },

    "projectId":  "<guidhere>"

}

]

Now I can run my imports:

terraform import azuredevops_agent_pool.pipelinesagent 9

terraform import azuredevops_agent_queue.pipelinesqueue <guidhere>/45

And a `terraform apply` shows no changes needed. 

ESXi 7 PCI Passthrough Gigabyte B550 board

 

System:

Gigabyte B550 Aorus Elite Pro

Ryzen 3100

Sandisk 32GB USB sticks

Problem:

After trying to pass through an LSI 9207-8i, subsequent reboots still showed the device in " Enabled / Needs reboot" in ESXi 7.0 and the device was not assignable.

I worked through a few steps:

Disabled the LSI driver so it wouldn't grab the card

esxcli system module set --enabled=false --module=lsi_msgpt2

(can't remember if it was gpt2 or gpt3 though)

Changed system UEFI settings 

Found these on reddit, not sure if they really helped: 

Tweaker -> Advanced CPU Settings -> SVM Mode -> Enabled
Settings -> Miscellaneous -> IOMMU -> Enabled
Settings -> NBIO Common Options -> IOMMU -> Enabled
Settings -> NBIO Common Options -> ACS Enable -> Enable
Settings -> NBIO Common Options -> Enable AER Cap -> Enable

Disable ACS Checks in ESXi

Manage > System > Advanced settings
Set VMkernel.Boot.disableACSCheck to True.

 

 

 Reboot. Done.

ESXI 7 installer error 28 vmx.v00 and resolution

ESXI 7 installer error on Ryzen 3100 machine

System:

Gigabyte B550 Aorus Elite Pro

Ryzen 3100

Sandisk 32GB USB sticks

 Seeing the following error during the boot of the install USB stick, imaged by Rufus from ISO.

CRC error during decompression, Received CRC (<snip>) != calculated CRC (<snip>)

gzip_extract failed for /vmx.v00 (size 0): CRC error

Error 28 (CRC) error while loading module: /vmx.v00

Compressed MD5: <snip>

Decompressed MD5: 0000000000000<snip>

Fatal error: 28 (CRC Error)

Tried:

• Different USB ports
• Different ESXI versions (6.7 produced similar errors) 
• Different USB sticks
• Different partition types (MBR vs GPT)
• Booting from an existing ESXI 6.7 install

Resolution found after these two steps: 

• Disabled CSM (Compatibility Support Mode) in the UEFI settings
• Ensured the machine booted from the UEFI bootloader on the USB stick. For some reason the BIOS wanted to boot from "VMWare USB FDMAP" or something, but there were two additional boot entries that were prefixed with "UEFI: " and that worked like a charm.

 

Searching for the CRC 28 error found zero helpful hits, so hopefully this will help provide relevant resolutions for someone.

Amazon Alexa App / Echo Dot "Failed to authorize link code. Please try again."

This is just consumer gear, but Googling didn't turn up anything at all for the error, so here you go.

Symptoms:
Try to set up a freshly reset Echo device. When joining WiFi, you receive an error in a red banner at the top of the screen that says "Failed to authorize link code. Please try again."

Resolution:
Try from another device running the Alexa app. I tried from my wife's phone (where the Echo Dot order was originally placed) and it worked straight away.

ESXi 6.7u2 update to 6.7u3 failed

Error on my host: Update manager fails to patch the host. When running the manual patch, the host profile update fails.

Error in esxupdate.log:

2019-08-25T00:45:24Z esxupdate: 2099348: root: ERROR: esximage.Errors.InstallationError: ('VMware_locker_tools-light_10.3.10.12406962-14141615', '[Errno 28] No space left on device')

Resolution:
cd /tmp
wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_6.5.0-0.23.5969300.vib
esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.10.12406962-14141615.vib

Resume manual patching:
esxcli software profile update -p ESXi-6.7.0-20190802001-standard \ -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml 

esxcli network firewall ruleset set -e false -r httpClient

reboot

Done.

DPM agent push to 2008 Fails 0x80990a1c

Install protection agent on node.node.com failed:


Error 347: An error occurred when the agent operation attempted to create the DPM Agent Coordinator service on node.node.com.


Error details: Unknown error (0x80990a1c)


Recommended action: Verify that the Agent Coordinator service on node.node.com is responding, if it is present. Review the error details, take the appropriate action, and then retry the agent operation.






Googling the error indicates that KB975759 may be missing.


This KB is not installed on the trouble server.






Manually installing the update MSU package says "update does not apply to your system".






Expand the update into a folder.


Expand –F:* c:\kb975579\Windows6.1-KB975579-x64.msu c:\temp\expandkb975579\






Run the installer (this takes a while!)


Start /w pkgmgr /ip /m:”c:\temp\expandkb975579\Windows6.0-KB975579-x64.cab






Reboot






Re-push the DPM protection agent.

ESXI 6.7 - Clone vmdk fails on change tracker error code 5948

vmkfstools -i /vmfs/volumes/ssd-ds1/filer/filer.vmdk /vmfs/volumes/ssd-ds2/filer/filer.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/ssd-ds1/filer/filer.vmdk'...
Clone: 100% done.Failed to clone disk: Invalid change tracker error code (5948).

Resolution:

1. rm the change tracker file (filer-ctk.vmdk)
2. vi /vmfs/volumes/ssd-ds1/filer/filer.vmdk , comment out the change tracker line. 
3. Repeat clone (vmkfstools -i) 

App Service Backup Fails when Including Azure MySQL DB

The application itself is running just fine on this DB string, but when including the Azure MySQL database in the App Service backup configuration, the backup fails.

Log Details:

MySql backup exited with code 2 for db:1cdda146. Output: mysqldump: Got error: 9002: The server name you tried cannot be found. Please use the correct name and retry. Please check your server name bdef0f98dd83a7ff16432ffc1c54394. when trying to connect

App Service Connect string:
Database=1cdda146;Data Source=bdef0f98dd83a7ff16432ffc1c54394.mysql.database.azure.com;User Id=c35e433d7a@bdef0f98dd83a7ff16432ffc1c54394;Password=<pw>

The DB name here is made up to match the example - the DB resource name is 31 chars. 

I can add another connection string with a different database - 29 chars, and that works. 

There also appears to be a lag in the App Service backup pulling the changes in the connection strings. If you change the settings for connection string N and immediately go run a backup, the old settings appear to be used. 

I've not found anywhere to point a finger at on this, yet. The log entry has a truncated server name. 

Next up I'll try making two new Azure MySQL DBs with 31 and 20 chars, to test my theory.

Update 1:
The new 31 char and 20 char DBs both back up fine. I guess I'll just make a new Azure MySQL DB and migrate schemas it.

Azure Application cannot enable SSL

I was testing out some stuff and ran across this error when trying to add an SSL cert to an application:

"Cannot enable SNI SSL for a hostname foo.bar.com because current site mode does not allow it."

This unhelpful error is just because you have selected a Free version of the Application Service plan. It has nothing to do with the web app you're in right now, other than the meager fact that you want to add SSL to it.

Upgrade the plan to a Standard or Premium plan to get past this error.

Unauthenticated SSH port forwarding in Cisco CSS 11500

References:
CISCO:   http://tools.cisco.com/security/center/viewAlert.x?alertId=37889
MITRE:  https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-0667

Timeline:
Feb 27 2015: Reported to Cisco PSIRT. Assigned to Incident Manager.
Mar 13 2015: Status check with Incident Manager.
[ Mar 18 2015: Cisco releases IntelliShield ID 37889 ] -- Not copied on this.
Apr 10 2015: Status check with Incident Manager
Apr 13 2015: Incident Manager supplies IntelliShield ID; Finding closed from my end.

mozjpeg findings update

Closed

20150323:  mozjpeg: b6029d31 SIGFPE on specially crafted jpeg https://github.com/mozilla/mozjpeg/issues/153

libicns

Findings

010db139 -- SEGV #1 icns_read_be (size=2, inp=0x661000, outp=<synthetic pointer>) at icns_io.c:51
3454826d -- SEGV #1 icns_read_le (size=2, inp=<optimized out>, outp=<synthetic pointer>) at icns_io.c:101

Reported to maintainer Mathew Eis  March 14, 2015, 9:43 p.m.

Fuzzing Update

freeXL:
Re: 9f74b0e8, c9be2aa7, d7273f72, 6889d18b
Closed with patch.
https://www.gaia-gis.it/fossil/freexl/fdiff?v1=2e167b337481dda3&v2=61618ce51a9b0c15&sbs=1

html2text:
Re: 8afd955e [Crash, SEGV ] malloc in Area.C:223
Response received from vendor, wontfix.

exifprobe:
Re: 3a41cb1c, 6bb4db60
Closed with patch. 3a41cb1c resolved in this patch. 6bb4db60 resolved in this patch.
Re: ab50ccf7
Closed in latest git checkout. Asking vendor for patch details.

fig2xsd:
Re: 485f75d6, bd644d4d
Closed with vendor release of 0.22
http://prdownloads.sourceforge.net/fig2sxd/fig2sxd_0.22.orig.tar.gz?download

gif2png:
Re: 0cd10b14
Vendor working on fix.

hp2xx:
Submitted new 0ad69d36, 24dfd181, 27eac1b7, 2b9f1f19, 3503093d, 3a5b6ff4, 3b2a6ca6, 40dd99c6, 42e898d9, 4fa145b4, 5570ae2a, 57dc6ada, 7c503fc7, 7d580ece, 87aebb39, 8d2e9b05, 9071d6d8, 91fe0b08, 92cee751, 99f38038, a921cb74, b1ee740e, b3445f4e, b6bd0cc0, b9eff3a0, bbb8391b, c3e9eae3, c7f0611a, cb3d04f9, d34f9134, d6ce68b4, dccd6129, ea388b76, f2315602, f7cede9e  on 3/8/2015

giftrans:
Submitted new a4ac94f0 to Debian QA team on 3/8/2015, Debian bug ID 780079;

Fuzzing Update

ImageMagick: 
Mutliple CPU DOS
http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26929
http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26931
http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26932
http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=26933

GraphicsMagick:
Cineon stack smashing and Fits assertion errors 1/24
http://www.graphicsmagick.org/Changelog.html

Mozjpeg:
SIGFPE on jpeg
https://github.com/mozilla/mozjpeg/issues/153

New CVE ID: CVE-2014-8716


 I found this via the AFL fuzzer.

sysidcfg in solaris 10 shared-ip zones

If you've tried to put a sysidcfg in shared-ip zones with multiple network interfaces, you've probably experienced some headaches.

I found that if I specified each interface like you would in a normal sysidcfg, it would drop to interactive input.

When I was just specifying one interface with network_interface=PRIMARY{etc etc}, everything worked fine. When I would switch to network_interface=e1000g0{primary hostname=server.local} , it would drop to interactive again.

The secret was just to use PRIMARY no matter how many interfaces I was configuring.

After some tinkering I was able to drop the whole network_interface{} stanza down to:

network_interface=PRIMARY{hostname=hostname.local}

Verify backups before implementing changes

Going to make a change to a machine?

Verify that you have good colds before you go through.

Almost every change or upgrade plan I have starts off with:

1. Day after colds: Verify backups for hosts {a,b,c,d,e,f,g,h} 
2. Day of implementation:  Verify backups for hosts {a,b,c,d,e,f,g,h}

If that backup isn't good, either take a good backup immediately before preceding or reschedule your implementation for a window after the next full backup.

I've had this save me a time or two. A previously bulletproof backup decided to cook off the week of one of my upgrades.

Oracle 10g on ZFS IOPS

I was reviewing a machine of ours that people had been complaining about. The disks were horribly busy and it made it difficult to tell what was hurting the performance in Oracle Apps. zpool iostat for the DB pool was constantly running about 4,000-6,000 read OPs at up to 65MB/sec, which seemed off. The ARC prefetch misses were really high and it occurred to me that ARC was prefetching at all. After disabling file-level prefetch the zpool iostat numbers dropped to 0-150 read OPs immediately. I really thought I had broken it, but I logged in to the apps and tested things out, everything was snappy.

Many thanks to ZFS Evil Tuning guide.