In a two-way relay network, two terminals exchange information over a shared wireless half-duplex channel with the help of a relay. Due to its fundamental and practical importance, there has been an increasing interest in this channel. However, there has been little work that characterizes the fundamental tradeoff between the communication reliability and transmission rate across all signal-to-noise ratios. In this paper, we consider amplify-and-forward (AF) two-way relaying due to its simplicity. We first derive the random coding error exponent for the link in each direction. From the exponent expression, the capacity and cutoff rate for each link are also deduced. We then put forth the notion of bottleneck error exponent, which is the worst exponent decay between the two links, to give us insight into the fundamental tradeoff between the rate pair and information-exchange reliability in the two-way relay network. As applications of the error exponent analysis to design a reliable AF two-way relay network, we present two optimization framework to maximize the bottleneck error exponent, namely: i) the optimal rate allocation under a sum-rate constraint and its closed-form quasi-optimal solution that requires only knowledge of the capacity and cutoff rate of each link; and ii) the optimal power allocation under a total power constraint and perfect global channel state information, which is shown equivalently to a quasi-convex optimization problem. Numerical results verify our analysis and the effectiveness of the optimal rate and power allocations in maximizing the bottleneck error exponent, i.e. the network information-exchange reliability.