The only easy case I can think of is where $\mathrm A_{ij} = \mathrm A_0$, where $\mathrm A_0$ is an invertible $n \times n$ matrix, for all $i,j \in [d]$. In this special case,
$$\mathrm A = \begin{bmatrix} x_{11} \mathrm A_{0} & x_{12} \mathrm A_{0} & \cdots & x_{1d} \mathrm A_{0}\\ x_{21} \mathrm A_{0} & x_{22} \mathrm A_{0} & \cdots & x_{2d} \mathrm A_{0}\\ \vdots & \vdots & \ddots & \vdots \\ x_{d1} \mathrm A_{0} & x_{d2} \mathrm A_{0} & \cdots & x_{dd} \mathrm A_{0}\\\end{bmatrix} = \mathrm X \otimes \mathrm A_0$$
where $\otimes$ denotes the Kronecker product. Hence,
$$\det (\mathrm A) := \left( \det (\mathrm X) \right)^n \cdot \left( \det (\mathrm A_0) \right)^d$$
and, thus, $\mathrm A$ is invertible if and only if $\mathrm X$ is invertible. If that is indeed the case, the inverse is
$$\rm A^{-1} = X^{-1} \otimes A_0^{-1}$$